Transcript: I built the image pipeline today, at least the base of it. One thing that it still needs is to embed, but beyond that, I think more or less it works, which is cool. So with the image pipeline, right now the steps are pretty simple. First, we go get some metadata from the EXIF information from the image, if it exists. I probably should, you know, not super rely on EXIF, but that's whatever for now, doesn't really matter. It's mostly photos that I care about. So all my photos are going in, the EXIF is getting processed. I'm specifically pulling out latitude, longitude, and the image creation time. I am then taking the latitude and longitude and reverse geocoding it and calling an LLM on top of it just to process the data just a little bit. And then after I've done that, I pass it into GPT for vision and I generate a caption for the image. And then after I've passed it into vision, I generate a title so it can go on the webpage easily. And really it's as simple as that. So that's kind of where we are today. And a few things learned along the way. One thing is like serving images in full size is annoying. And also I am hesitant to build an entire like image downscaling, rescaling pipeline. That seems like a lot of work. So figuring out how to host this is gonna be very important very quickly. However, for now it doesn't matter. There are some other things that I'm thinking about. One being like discovered along the way, one of the things that I'm curious about is like should I just be storing a bunch of data in the metadata.json file, like any metadata that's collected and giving that to GPT to figure out what to do with. Specifically like the EXIF information contains a lot of interesting things that like if I wanted to build a photography page, like I might want the shutter speed and aperture, right? And that's stuff that I'm not covering right now. The other things are thinking about the pipeline process and thinking about each of these steps as almost completely independent and very much kind of like, I guess OpenAI's function calls in some way. But more like val.town where these things just output something and then you might do another thing with that. And generally like the order is probably deterministic and you know what it is going to be at least on the ingestion step. But also this could be helpful for LLMs to begin to program the steps itself. And it also passes the data off from a central location to a bunch of microservices, which could be useful, could be not useful. Really depends on the specific user and use case, I think. And that's also something I'm thinking about as I'm starting to depend on more and more APIs and X and Y. It's a lot to set up, it's a lot to set up, so like really need the basic things to work really well to show that it's worth setting these things up, yeah. And then there's the thing of collapsing the pipelines and making them common. I already did metadata across audio and images that should share the same step, but it's just not very clean right now, so it is kind of feeling like you might want to have like a step library. These are the library of like little small utilities that you can use, assuming everything else works nicely.
The author discusses the need to group individual steps in composing pipelines and seeks advice on existing products from Jamie. They express the goal of improving the infrastructure for Glyph but acknowledges the current lack of resources. They emphasize focusing on the problem and making the execution of LLMs faster, and the ability to experiment with them quickly. Their ultimate aim is to understand human context and establish protocols between AI agents, while also streamlining the architecture and recording context.
87.54% similar
Today marked a significant advancement in the burrito project, where the image pipeline, established the previous day, became fully functional and integrated into a webpage, complete with an effective querying system. The visual aspect of the project, particularly the image embeddings, was both intriguing and aesthetically pleasing, although its effectiveness is still under review. The project is now at a stage where the creator is keen to move beyond personal experiments to sharing the results with others, with the immediate goal being to encourage a small group of individuals to test the developments. The focus for the week has shifted to actual user engagement through getting people to sign up and provide feedback, driven by the enthusiasm of witnessing the project's imagery features come to life.
87.42% similar
The individual has discovered that working backward from a desired result with a large language model is surprisingly effective, especially when detailing the problem forward seems challenging. This backward approach has simplified the problem and resulted in the use of GPT-4 for data transformation within the context window, improving the process. An automatic metadata generation pipeline is emerging, where data transformations are added as needed, potentially storing transformations for future use based on query relevance. This system will generate an extensive amount of synthetic data, allowing for the extraction of relevant information through queries fed into the model at later stages, rather than having to pre-determine all questions.
In the first bucket, the focus is on achieving AI-level parallelism, creating a better pipeline, enabling the execution of different LLM tasks in parallel, and allowing future agents to add information to an execution graph. This parallelization is crucial for distributed systems processing and likely to advance the distribution and parallel running of models. The second bucket involves implementing transformations, such as converting unstructured transcripts into organized bullet point lists, and making this adaptable and viable through JSON. The goal is to seamlessly convert text into a GitHub issue, providing instructions for transformation and capturing context to refine models.
The speaker aspires to be part of communities that empower individuals to explore their data and bring value back to themselves. They are willing to take a job in such a space and believe it's worth doing. The goal is to build tools that make it easy for the individual to work with their data directly on a web page. They plan to move to a more reactive front end using Next.js and React, designing a feed and query system possibly using natural language. The speaker also mentions working on embedding audio and ensuring embeddings are accessible. The text discusses the process of obtaining and manipulating data and emphasizes the importance of experimentation and innovation. It uses the metaphor of building a playground to illustrate the iterative nature of the process, acknowledging that initial attempts may be imperfect but can be improved upon through learning from mistakes. The writer anticipates challenges but expresses a hope to avoid negative consequences and eventually achieve success. Finally, the text concludes with a lighthearted remark and a reference to going to sleep.