Transcript: It's funny, with the large language model stuff, working backwards from the answer that I want has seemed to help more than trying to work forward from the large language model, which is very interesting in terms of how it reasons, where, like, I'm working backwards through a problem and it has to, like, write things down on a piece of paper before it continues, which is really fascinating to think about the differences, like, it's hard for me to work this problem forwards, actually. I mean, like, I can just intuit about it forwards, but to then describe it effectively to a large language model, I'm working it backwards. Interesting. But I think, I mean, made decent progress at the very least. I think the problem has been simplified by working backwards in this way. So the last step of the pipeline is just, like, do a data transformation, more or less. And initially I was using, like, a small stupid model to do that, and instead it's just like, okay, we're just going to use GPT-4 and shove everything in the context window under a certain size and do that, and that'll be good enough. And indeed that's what we're going to do. So given that, like, it's helping me start to figure out some aspect of training data as well, like what data I will need to train. And then also it is effectively generating tons of synthetic pairs. So I will want to generate individual synthetic pairs as well. This is something that I'm thinking about after this next step gets done effectively. After this step is done, well, then we can go and basically what we'll want to do is remove whatever transform we're doing. So if we've transformed some data, we're going to want to go back to the metadata field, look at the unstructured data, and just immediately run that transformation on that piece of data. And if there is nothing to be transformed, we'll just output null, and nothing will go back into the metadata file. But this is effectively going to be automatically adding steps to the pipeline. That's kind of what's happening here. You ask it a question, and it's going to go and just add that data to the pipeline. Initially, none of that data is going to be used probably, but in the future, what would be great is to effectively cache those transformations and have a description of what the transformations that you have are on any piece of data. And if that transformation matches up with the user's query in any reasonable way, and again, maybe this is using embeddings or something, or maybe just a large language model that's tuned to it, it goes and does that. But this is going to be effectively like an automatic metadata generation system that will hopefully continuously improve. And we're just going to generate a massive amount of synthetic data over whatever data we put in. And I don't need to come up with all of the questions for the data in advance, because I can go and ask those questions later to get the relevant data I want out of it. And that is going to be kind of what we are really doing initially.
The speaker aspires to be part of communities that empower individuals to explore their data and bring value back to themselves. They are willing to take a job in such a space and believe it's worth doing. The goal is to build tools that make it easy for the individual to work with their data directly on a web page. They plan to move to a more reactive front end using Next.js and React, designing a feed and query system possibly using natural language. The speaker also mentions working on embedding audio and ensuring embeddings are accessible. The text discusses the process of obtaining and manipulating data and emphasizes the importance of experimentation and innovation. It uses the metaphor of building a playground to illustrate the iterative nature of the process, acknowledging that initial attempts may be imperfect but can be improved upon through learning from mistakes. The writer anticipates challenges but expresses a hope to avoid negative consequences and eventually achieve success. Finally, the text concludes with a lighthearted remark and a reference to going to sleep.
In envisioning an ideal way to integrate new log entries, the goal is to place each entry within the larger context of the whole, which may be an iterative process to determine that context. The author contemplates whether incorporating various data sources into a language model like GPT-4 could help it understand the overarching themes of communications, such as text messages. They propose an experimental approach by loading as much context as possible into the model whenever a new input is received, maximizing the token limit to allow the model to contextualize new information based on previous entries. This method, which involves brute forcing context into the AI's understanding, could potentially be a valuable asynchronous step in refining the pipeline for more nuanced contextual analysis.
The speaker is considering the research question of how to achieve distributed compute, particularly the need for parallelism in executing pipelines and AI agents. They question the potential for building a Directed Acyclic Graph (DAG) that allows for agents to dynamically contribute to it and execute in parallel, emphasizing the need for pipeline development to accommodate this level of complexity. The discussion also touches on the scalability and parallel execution potential of the mixture of experts model, such as GPT-4, and the potential for hierarchical or vector space implementation. The speaker is keen on exploring the level of parallelism achievable through mixture of experts but acknowledges the limited understanding of its full capabilities at this point. They also express curiosity about fine-tuning experts for personal data. The speaker is discussing the data they are generating and the value of the training data for their system, particularly emphasizing the importance of transforming the data to suit their context and actions. They mention meditating and recording their thoughts, which they intend to transform into a bullet point list using an AI model after running it through a pipeline. The individual also discusses making their data publicly accessible and considering using GPT (possibly GPT-3) to post summaries of their thoughts on Twitter. They also ponder the potential of using machine learning models to create a personal Google-like system for individual data. The text discusses using data chunking as a method for generating backlinks and implementing PageRank in an agent system. It mentions steep space models and the continuous updating of internal state during training. It also compares the level of context in transformer models and discusses the idea of transformer as a compression of knowledge in a language. The speaker expresses interest in understanding the concept of decay in relation to memory and its impact on the storage and retrieval of information. They draw parallels between the processing of information in their mind and the functioning of a transformer model, with the long-term memory being likened to a transformer and short-term memory to online processing. They speculate on the potential of augmenting the transformer model with synthetic training data to improve long-term context retention and recall. Additionally, they mention a desire to leverage a state space model to compile a list of movies recommended by friends and contemplate the symbiotic relationship between technology and human sensory inputs in the future. In this passage, the speaker reflects on the relationship between humans and computers, suggesting that a form of symbiosis already exists between the two. They acknowledge the reliance on technology and the interconnectedness of biological and computational intelligence, viewing them as mutually beneficial and likening the relationship to symbiosis in nature. They express a preference for living at the juxtaposition of humans and computers, while acknowledging the potential challenges and the need to address potential risks. Additionally, they mention that their thoughts on this topic have been influenced by their experiences with psychedelics. The speaker discusses the potential increase in computing power over the next five years, mentioning the impact of Moore's Law and advancements in lithography and semiconductors. They refer to the semiconductor roadmap up to 2034, highlighting the shift towards smaller measurements, such as angstroms, for increased transistor density. They emphasize that the nanometer measurements are based on nomenclature rather than actual transistor size, and the challenges in increasing density due to size limitations and cost constraints. The conversation touches on different companies' approaches to transistor density and the role of ASML in pushing lithography boundaries, before concluding with a reference to the high cost and potential decline in revenue for semiconductor production. The speaker discusses the importance of semiconductor manufacturing in the U.S. and China's significant focus in this area. They mention watching videos and reading sub stacks related to semiconductor technology, specifically referencing industry analysts and experts in the field. The speaker expresses enthusiasm for staying updated on developments and offers to share information with the listener. The conversation concludes with a friendly farewell and the possibility of future discussions.