Deploying a small Japanese LLM on Cloud Run

Introduction

This article explains in detail how to deploy the small Japanese language model “TinySwallow 1.5B” on Cloud Run and build a cost-efficient, practical LLM application. GPU cost is a major challenge when operating large language models, but Cloud Run’s serverless architecture keeps costs to a minimum during periods with no traffic.

If you want to see the code right away:

What small language models can do, and why this one

Recent small language models perform surprisingly well, and for certain tasks they are nearly as useful in practice as large models. This implementation uses “TinySwallow 1.5B”. TinySwallow is a small Japanese model developed by Sakana AI. The same approach also works with other small models such as Gemma-3 and Mistral-7B, but as a model specialized for Japanese, TinySwallow is a very attractive choice.

System architecture in detail

The system uses a distributed architecture made up of the following two main components:

Frontend/backend server (Go)
- Serves the user interface
- Handles API requests
- Communicates with the LLM service
LLM service (Ollama + TinySwallow)
- Runs the language model
- Performs inference

System architecture diagram

This separated architecture has the following benefits:

Scalability: Each component can scale out independently
Resource optimization: The frontend runs on minimal resources, and only the LLM part gets high-spec instances as needed
Development flexibility: The model part alone is easy to update or replace

Frontend tech stack

The frontend is built with the following technologies:

Go standard library html/template: Template rendering
htmx: Dynamic UI with minimal JavaScript
CSS: Simple, lightweight styling

Rather than building a full SPA (Single Page Application), this setup delivers a fast experience through simple HTTP requests and partial HTML rendering.

Implementation details and best practices

Packaging the model efficiently with Docker

To distribute and start TinySwallow efficiently, a Docker image that already contains the model is built in advance. With this approach, containers do not have to wait for the model to download at startup, which greatly reduces cold start time.

Best practices for region selection and GPU use

When using GPUs on Cloud Run, keep the region constraints in mind:

GPU-supported regions: As of March 2025, Cloud Run with GPUs is not available in asia-northeast1 (Tokyo)
Recommended region: Choose a region where GPUs are available, such as asia-southeast1 (Singapore)
CPU mode: Small models like TinySwallow also run on CPUs, so consider CPU mode if cost is the priority

Mitigating cold starts

Because of how Cloud Run works, cold starts when instances scale up from 0 are unavoidable. The following measures improve the user experience:

Minimum instance setting
- Keeping a minimum number of instances running at all times reduces the impact of cold starts
Pre-warming:
- Run health checks periodically to warm up instances

Detailed design of the Go backend

Project structure

The backend design follows the ideas of Clean Architecture:

.
├── cmd/
│   └── coda/
│       └── main.go         # Entry point
├── internal/
│   ├── config/             # Configuration management
│   ├── frontend/           # UI
│   ├── infrastructure/     # Infrastructure
│   └── llm/                # LLM integration
├── Dockerfile

Integrating with the LLM service

The integration with external LLM services such as Ollama is designed for readability and maintainability. An LLM client interface is defined and the Ollama client implements it, which makes it easy to switch to a different LLM service in the future.

Error handling and retry strategy

In production, the connection to the LLM service is not always stable. Error handling and a retry strategy make the system more robust. A retry mechanism with backoff gets past temporary connection problems.

Building the CI/CD pipeline

For an efficient development cycle, the project uses a CI/CD pipeline that combines GitHub Actions and Google Cloud Build. Automating each step of testing, building, and deploying improves both development efficiency and quality.

Performance optimization and tuning

This section covers the settings that optimize performance in production:

1. Memory and CPU optimization

For TinySwallow 1.5B, the following settings are sufficient:

CPU: 8 vCPU
Memory: 32GB
GPU: NVIDIA L4 GPU x1

2. Tuning inference parameters

When running the model with Ollama, the main parameters for balancing inference quality and speed include temperature, top_p, top_k, and max_tokens. Tuning them for your use case gives the best results.

Cost estimate

Using a GPU requires allocating at least 8 vCPUs and 32 GiB of memory.

Detailed cost estimate (as of March 2025)

For example, running Cloud Run with one GPU for 8 hours (28,800 seconds) costs the following:

GPU cost: $0.0002796 × 28,800 seconds = $8.05248
CPU cost: 8 vCPU × $0.00002160 × 28,800 seconds = $4.97664
Memory cost: 32 GiB × $0.00000240 × 28,800 seconds = $2.21184

Total cost: $15.24096 (about 2,286 yen at 150 yen per dollar)

Monitoring and operations

A monitoring strategy for effective operations makes it possible to detect and respond to problems early:

Logging strategy

Structured logging makes problems easier to diagnose and analyze. Including information such as request IDs, latency, and token counts makes it possible to track and optimize performance.

Security best practices

Security always comes first. Pay particular attention to the following points:

1. Securing service-to-service communication

Use TLS encryption to protect communication. Always use HTTPS and implement appropriate authentication mechanisms to keep communication secure.

2. Preventing prompt injection

Always sanitize input to the LLM to prevent malicious prompt injection. In addition to basic sanitization such as length limits, clearly separating the system prompt from user input strengthens security.

Conclusion: pros and cons of running LLMs with Go

Pros

Concise codebase: Go’s clear syntax and standard library make the code highly maintainable
High performance: A low-latency, resource-efficient runtime
Excellent concurrency: Efficient streaming with goroutines
Easy deployment: A single binary minimizes dependency problems
Scalability: Works well with Cloud Run and scales automatically with load

Cons

Few ML libraries: Compared with Python, libraries for specialized ML processing are limited
Complex ML processing is hard: Deep learning workloads tend to be complex to implement
Ecosystem maturity: The Go ecosystem for LLMs is still developing

Lessons from production

Cost optimization matters: Even for small models, a serverless architecture brings large benefits
Put user experience first: Make technical issues such as cold starts visible and address them
Improve incrementally: Start with a simple implementation, then optimize based on data

References

I hope this article helps you run small language models effectively on Cloud Run. If you have questions or feedback, please open an issue in the GitHub repository.