Picture this: you’ve built a blazing fast Go service, only to find it moves like a sloth through peanut butter when talking to your NoSQL database. Fear not! Today we’re diving into professional-grade optimizations that’ll make your database interactions smoother than a jazz saxophonist’s riff. I’ll share battle-tested techniques and a few “ohhh, that’s why!” moments from my own coding misadventures.
Taming the Connection Beast 🔗
Let’s start with the foundation - connection management. Think of database connections like pet cats: too few and they get overwhelmed, too many and they’ll trip you constantly. Here’s how to configure a MongoDB connection pool properly:
import (
"context"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func createPool() *mongo.Client {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
clientOpts := options.Client().ApplyURI("mongodb://localhost:27017").
SetMinPoolSize(10).
SetMaxPoolSize(100).
SetMaxConnIdleTime(5 * time.Minute)
client, err := mongo.Connect(ctx, clientOpts)
if err != nil {
log.Fatal("Connection failed faster than my New Year's resolutions")
}
return client
}
Key numbers to remember:
- Min pool size: Keep 5-10 warm connections ready
- Max pool size: Don’t exceed 150% of your database’s max connections
- Idle timeout: 5-10 minutes for most applications True story: I once saw a service create 10k connections in 2 minutes. The DBA looked like they’d seen a ghost. We added pool limits and performance improved by 40%!
Batch Operations: The Art of Bulking Up 💪
Single document inserts are like using a teaspoon to empty a swimming pool. Be smart with bulk writes:
func bulkInsertUsers(users []User) {
models := make([]mongo.WriteModel, len(users))
for i, user := range users {
models[i] = mongo.NewInsertOneModel().SetDocument(user)
}
opts := options.BulkWrite().SetOrdered(false)
result, err := collection.BulkWrite(context.TODO(), models, opts)
if err != nil {
log.Printf("Bulk insert failed: %v", err)
}
log.Printf("Inserted %d users", result.InsertedCount)
}
Pro tip: Set ordered=false
unless you need strict sequence. This allows parallel document insertion and can give 3-4x speed improvements.
Indexing: The Dark Art of Query Speed 🧙♂️
Poor indexing makes databases work harder than a college student during finals week. Let’s create smart indexes:
// Create compound index with partial filter
indexModel := mongo.IndexModel{
Keys: bson.D{
{Key: "last_name", Value: 1},
{Key: "created_at", Value: -1}
},
Options: options.Index().
SetPartialFilterExpression(bson.M{
"status": "active",
}).
SetExpireAfterSeconds(86400 * 30), // TTL
}
_, err := collection.Indexes().CreateOne(context.TODO(), indexModel)
if err != nil {
log.Fatalf("Index creation failed: %v", err)
}
Golden rules for indexing:
- Profile your queries with
Explain()
- Favor compound indexes over single-field
- Use partial indexes for filtered queries
- TTL indexes are your friend for ephemeral data
The Monitoring Gauntlet 🔍
You can’t optimize what you can’t measure. Implement this monitoring stack:
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
queryDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "nosql_query_duration_seconds",
Help: "Time taken for database operations",
Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1},
}, []string{"operation", "collection"})
)
func instrumentedFind(filter bson.M) (*mongo.Cursor, error) {
start := time.Now()
defer func() {
queryDuration.
WithLabelValues("find", "users").
Observe(time.Since(start).Seconds())
}()
return collection.Find(context.TODO(), filter)
}
Key metrics to watch:
- Query duration percentiles
- Connection pool usage
- Error rates by operation type
- Cache hit ratios
The Cache Caboose 🚂
When all else fails, cache like there’s no tomorrow. But do it right:
type CachedUserLoader struct {
cache *ristretto.Cache
ttl time.Duration
}
func (l *CachedUserLoader) GetUser(id string) (*User, error) {
if val, ok := l.cache.Get(id); ok {
return val.(*User), nil
}
user, err := fetchFromDB(id) // Actual DB call
if err != nil {
return nil, err
}
l.cache.SetWithTTL(id, user, 1, l.ttl)
return user, nil
}
// Initialize cache with 10MB max size
cache, _ := ristretto.NewCache(&ristretto.Config{
NumCounters: 1e6,
MaxCost: 10 << 20,
BufferItems: 64,
})
Cache invalidation strategies:
- TTL-based for time-sensitive data
- Write-through on updates
- Bloom filters for negative caching
Final Showdown: Putting It All Together 🚀
Remember that service that crashed our DB? After applying these techniques:
- 90th percentile latency dropped from 2.1s → 127ms
- Connection errors reduced by 98%
- Cloud bill decreased by $4,200/month The moral? Database optimization isn’t just about tweaking configs – it’s about understanding the entire data access lifecycle. Now go forth and make your queries fly! Just don’t forget to come up for coffee breaks occasionally (we’re optimizing humans too, right?).