A sophisticated Streamlit application that performs targeted image editing using a "Detect-then-Segment-then-Inpaint" pipeline. By combining state-of-the-art computer vision models, this tool allows users to describe an object to change and then describe what it should become.
-
Natural Language Detection: Uses GroundingDINO to find objects in your image based on simple text prompts (e.g., "the blue shirt").
-
Precision Masking: Leverages Meta's Segment Anything Model (SAM) to generate pixel-perfect masks for the detected objects.
-
High-Fidelity Inpainting: Utilizes Stable Diffusion XL (SDXL) 1.0 to transform the masked area while maintaining lighting and texture consistency.
-
Dual-GPU Optimization: Intelligent device mapping across cuda:0 and cuda:1 to handle heavy model loads.
-
DeepCache Integration: Uses DeepCacheSDHelper to accelerate inference speeds without significant loss in quality.
To run this project install dependecies first
# Core libraries
!pip install diffusers accelerate safetensors transformers DeepCache streamlit opencv-python
# Segment Anything
!pip install scipy segment_anything
# GroundingDINO
!git clone https://github.com/IDEA-Research/GroundingDINO.git
cd GroundingDINO
pip install -e .
cd ..🧠 Download Model Weights You will need to download the pre-trained weights for the vision models:
mkdir weights
cd weights
# GroundingDINO weights
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
cd ..
# SAM weights
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pthRun the application:
streamlit run my_app.py-
Upload an Image: Drop a JPG or PNG file into the uploader.
-
Define the Object: In the Object Selection Prompt, type the name of the item you want to change (e.g., "sunglasses").
-
Describe the Change: In the Image Manipulation Prompt, describe the new look (e.g., "futuristic neon goggles").
-
Adjust Settings:
-
Guidance Scale: How closely the AI follows your text prompt.
-
Strength: How much of the original masked area should be rewritten.
-
-
Submit: View the detection box, the mask, and the final high-resolution result.
Gemini said To make this look professional in a .md (Markdown) file without relying on external rendering (like Mermaid), the best approach is a combination of a clean table and a flowchart-style list.
You can copy and paste the section below directly into your README.md:
-
Input: User uploads an image and provides two prompts (Selection & Modification).
-
Locate: GroundingDINO scans the image to find the coordinates of the object requested.
-
Isolate: SAM takes those coordinates and creates a surgical-grade mask of the object’s shape.
-
Transform: SDXL uses the mask to replace the object based on the second prompt while maintaining the original background.
-
Optimize: DeepCache ensures the generation is fast, and flush() clears the VRAM for the next task.
-
Object Selection: Be specific if there are multiple items (e.g., "the car on the left").
-
Manipulation: Mention lighting and style to help the AI blend the new object (e.g., "A red leather jacket, cinematic lighting, 8k").
-
Multiple Objects: The app supports multi-masking; just separate your objects with commas and toggle the "Multiple" radio button.
-
Content Filtering: The app includes a built-in keyword filter to prevent the generation of explicit or harmful content.
-
Memory Management: Includes a flush() utility to clear GPU VRAM and cache after every generation, preventing Out-of-Memory (OOM) errors during long sessions.
