Show HN: We scaled Git to support 1 TB repos https://ift.tt/QT1jBPw

By
Show HN: We scaled Git to support 1 TB repos I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://ift.tt/szN9g8n Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client & cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo. XetHub is available today (Linux & Mac today, Windows coming soon) and we would love your feedback! Read more here: - https://ift.tt/cb25Pqm - https://ift.tt/b3tp0rA https://ift.tt/5dF9eSU December 13, 2022 at 07:14AM

Comments

Post a Comment

Popular posts from this blog

Jennifer Garner's Signature Fall Salad

By
Image
My family made this for Thanksgiving years ago and we ended up liking it so much it went into the rotation. I love when vegetables take center stage and it feels like a good thing, and not [like] when I was kid and it was a bummer! Thanks to the vinegar and maple syrup in the dressing, the vegetables come to life. Prep the vegetables — in batches! The first two to go into the oven are sweet potatoes and carrots. READ MORE... from Kitchn | Inspiring cooks, nourishing homes https://ift.tt/VnJeHTu
Read more

Trisha Yearwood’s Famous Blueberry Crumb Bars Are So Good, I’m Making Them on Repeat This Summer

By
Image
Summer is just a few weeks away, and I’m on the lookout for ways to put a fresh spin on classic desserts. One treat I always love during the warmer months? Blueberry pie. So when I came across a popular recipe (boasting hundreds of five-star reviews!) that reimagines the classic, I knew I had to give it a try.This remixed blueberry pie recipe comes from singer and Food Network star Trisha Yearwood, who transforms the traditional summer staple into a custardy bar. The reviews are glowing. READ MORE... from Kitchn | Inspiring cooks, nourishing homes https://ift.tt/9ZjkARQ
Read more

French Apple Skillet Custard Cake

By
Image
The rush to work, school, or sports means breakfast is often an egg burrito or a jar of overnight oats on the way out the door. The tempo slows at the end of the weekend, so I’ve dubbed it “Skillet Sunday.” Breakfast on this day is served hot and fresh, but the energy behind it is relaxed for everyone — including the cook. Think: skillet pancakes baked at once instead of a stack of pancakes flipped one by one. READ MORE... from Kitchn | Inspiring cooks, nourishing homes https://ift.tt/LcNfEzb
Read more