aimanhaziq/gemma4-2b-coder-GGUF

Gemma 4 2B Coder (GGUF)

Quantized GGUF binary of aimanhaziq/gemma4-2b-coder for fast, lightweight local and edge inference via llama.cpp, Ollama, LM Studio, and Open-WebUI.

• Base Architecture: Google Gemma 4 (Effective 2B)
• Primary Specializations: Production Kubernetes manifests (Deployments, StatefulSets, Ingress, CRDs), idiomatic async Rust (Tokio, Axum, ownership patterns), and TypeScript/Web API backends.
• Quantization: Q4_K_M (4-bit Medium: optimal trade-off preserving attention layer precision while minimizing footprint).

---

Model Files & Specifications

Filename	Quant Type	File Size	Memory (VRAM / RAM)	Best For
gemma4-2b-coder-Q4_K_M.gguf	Q4_K_M	~1.8 GB	~2.5 GB peak	Edge devices, macOS Metal, low-spec cloud VMs, mobile

---

1. Run with Ollama

Step 1: Create a Modelfile

Save the following as Modelfile in the folder where your .gguf file is stored:

FROM ./gemma4-2b-coder-Q4_K_M.gguf

TEMPLATE """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{{ .Prompt }}

### Input:
{{ .System }}

### Response:
"""

PARAMETER stop "<end_of_turn>"
PARAMETER stop "### Instruction:"
PARAMETER stop "### Response:"
PARAMETER temperature 0.2
PARAMETER top_p 0.95
PARAMETER num_ctx 8192

Step 2: Build and Execute

# Build the local Ollama model
ollama create gemma4-coder -f Modelfile

# Run interactive prompt
ollama run gemma4-coder "Write a Rust function using Tokio to query a Kubernetes metrics endpoint."

---

2. Run with llama.cpp

Interactive CLI (llama-cli)

./llama-cli \
  -m gemma4-2b-coder-Q4_K_M.gguf \
  -p "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a production-ready Kubernetes NetworkPolicy that restricts inbound traffic to port 8080 from pods labeled app: frontend.\n\n### Input:\n\n\n### Response:\n" \
  -n 512 \
  -c 8192 \
  --temp 0.2 \
  -ngl 99

(Use -ngl 99 to offload all layers to your GPU/Metal engine, or omit it to run entirely on CPU).

Standalone OpenAI-Compatible Server (llama-server)

./llama-server \
  -m gemma4-2b-coder-Q4_K_M.gguf \
  --port 8080 \
  -c 8192 \
  -ngl 99 \
  --host 0.0.0.0

---

3. Training & Quantization Details

• Instruction Fine-Tuning Format: Alpaca template (### Instruction:, ### Input:, ### Response:).
• Quantization Toolchain: llama.cpp (convert_hf_to_gguf.py -> llama-quantize).
• Context Length: Supports up to 8k native tokens.