Transcript: No. Okay. Yeah, totally. Yes, yes. Yes, yes. Yeah. Well, I think that's basically that's basically exactly what I'm thinking is like there's general research question how to get there is basically unknown. That's that's that's basically that's basically the insight that is that are like that's what I feel is like how to get there is definitely unknown and just building. So the experimental testbed how I'm thinking about it like in it in and of itself like part of the thing that I'm thinking about it is from a distributed compute perspective because well, well it kind of kind of yes is so I have I have two things that one is like from distributed compute perspective like one of the things that I'm doing by building pipelines is basically being able to see where parallelism can happen is like I'm already building these pipelines by hand and it just executes, you know singularly and I know with glyph you guys are kind of running in similar similar ways like that graph can be executed in parallel and I suspect especially with AI agents. Well, I want to be able to build a DAG and then also have agents be able to add to the DAG on the fly and I want AI level parallelism, you know, as if we have instruction level parallelism in CPUs. I want to be able to be executing AI's in parallel with each other, but their outputs may depend on other outputs. So how do we basically build a pipeline to make that happen? And that's like one of the one of the problems that I'm trying to solve at the moment because I think I think it's just going to be necessary. Once you get to a certain level of complexity is to be able to run things very very parallelly very parallelly and it also works very well for distributed compute in that in that case. And yes, and part of that part of that is as far as I'm aware like GPT-4 is based on mixture of experts and I'm very curious to see like to what degree mixture of experts scales and can you build quite compressed models as a result of that? I don't know but I it is it is very curious to me and if that's the case maybe there is like some again like high degree of parallelism that can be achieved through the mixture of experts kind of model effectively as well. Like it seems like it's so new that we actually just don't know yet but I wonder if there's ways to parallelize that much further and I think that's yeah. I had something else but I forget what it was. I mean I think I think higher I I think like to some degree like kind of kind of I think in fact it to me it seems like it almost might be hierarchical. I'm not sure it's possible that you could do it in like you know another another like vector space effectively but as far as I'm aware like with the mixture of experts is like you have kind of have like one gate at the top that's like pick the two most relevant experts and then they're figuring out how to generate their own models. And then they're figuring out how to generate the tokens. However I'm assuming it's like some basic like sigmoid or something to actually like figure out the next token. But I am very curious like for personal data like can I fine tune experts like effectively enough on my own data like generating synthetic fine tuning data and experts on each of those things like I don't know that's. Very curious to me. I it's like one of the things like with the data that I'm generating that I'm just curious about is like, well, in fact, like all of my inputs into this system like any transformations that I want to put on the data because that's more relevant to me is extremely extremely valuable training data because I am teaching it my context through my actions. Right is it right before this call? I meditated for 15 minutes and what I've been doing for meditating is also starting a voice recording at the same time and saying the things out loud in my head so I can move on from them, right? But like this is this is a I'm loving it so far and I also have like a thing that's I'm running my pipeline over that transcription. So now I'm getting now I'm getting a transcript of it. And now I already know because I meditated the way that I want to view that is in a bullet point list because those things are probably almost entirely unrelated or they can be nested within each other like like guaranteed. I don't want to look at a raw transcript a summary is going to be complete garbage like but a bullet point list of the transcript that transformation is perfect, you know, like that's actually how I want to see that data and by me like literally writing the transformation and just being like hey GPT for can you turn this into a bullet point list and nest things as necessary. It'll do it and it provides like it should be pretty good. I started it seven months ago. It's only recording my voice because I have headphones in I can I can pull up pull my headphones out and get you as well. But yes. Drinking your own Kool-Aid yeah or dog fooding and this whole thing and and and and also all of all of this data data right now that I'm putting in is like totally publicly accessible on the clear web and I just have it under a URL that like probably no one will find but like I it I can definitely share it with you. If you're interested, it's like it's pretty simple. It's just like summaries of whatever I'm thinking about and and the raw transcript from it. And that's that's the whole web page. That's just a bunch of text. Yes. Well, and and that's that's kind of where I'm trying to go with it is like I'm like the next the next thing that I really want to like build on the web page is basically this like little transformations thing is like I already have all the metadata. Yeah. I have a pipeline that's you know, it's like eight steps for audio files or whatever and now it's just like, okay, let me just pull, you know, like make that thing more visible to other people, you know, like I do think GPT generally is quite good at that. So at least to some degree and even looking at the summaries that are generated. I'm like half debating just adding a pipeline step at the end of it at the end of the pipeline that just like post the summary to Twitter and a link to like the raw page like, you know, like it would be very easy to do that. Yeah. Yes. Yeah. Oh, sure. Yeah. Yes. Yeah. Sure. Sure. Yes. It kind of is. It's it's it's it's interesting as you're talking about this. It's it's reminding me of the search and learning thing again and and being like I wonder if like some framing that really I'm going for is like less on the learning side and just being like we can take advantage of learning to basically build personal Google for everyone like it wouldn't be that hard or like theoretically not that hard. Like can you actually just take the model of PageRank and and apply it to your own personal data? I don't know. It's like and use use machine learning models to just like make that really easy instead of you know, like anything anything like super crazy. This is just like I don't know pass it through now and call it good enough. And like, you know, um, chunk the data and use that as a, as a method of generating backlinks and then using PageRank over it and then feeding, using that, using that in an agent system. Yeah. Yeah. Yeah. Yeah. Okay. Steep space models. Oh, this is Mamba. Okay. Okay. Okay. Yeah. Yeah. Hmm. Hmm. Interesting. Captured as part of that state. Yeah. I see. I see. It's like, it's, it's, it's like you like in the, in the sense, like, yeah, it's, it's, it's like you, you get it to some initial state as part of the training, but then after the training, as you're feeding through it through, if you just keep it live the whole time and you don't like kill the session, like effectively, like it's, it's just continuously updating its internal state as a result. Yes. Huh. That's fascinating. Yeah. Yeah. Intuitively feels much closer. It's, it's more, it's more agentic. And it also like, in terms of the things that I'm talking about with context, it seems like that has much higher degree of context than like having to just kind of shove relevant context into a transformer model and call it like good enough and their context windows is theoretically get so big that like it just works, but like, I don't know, seeing how big the context windows, they don't work very well as far as I I've seen, like it may be able to fit all of that, but it loses a lot of context as a result. Yes. Okay. Yeah. Oh, whoa. Okay. Okay. Hmm. Yeah, totally. What? Yeah. Yeah. Right. Right. Like, like how I viewed, how I view a transformer is like a compression of knowledge that speaks English or like whatever language, like that's more or less like how I, how I view it. Yes. Yes. Yes. Yes. Yes. Yes. Where, where is. This is like it is it is also compressed but in quite a different way I also I also wonder like through time like what decay is like in terms of like memory basically they're like memory it'd be interesting 100% I mean that makes sense huh interesting this is this is kind of reminding me of like to almost like two different paradigms in some sense like even in our own head of like transformer almost feels more like long-term memory and this seems a little bit closer to like short-term memory like online processing versus like you can take that online processing and then like then later like shove it into the transformer model as well totally well what I but but but but the thing is like well it's well I guess it again it depends on like what the decay is like so it may have a really long context window or effectively infinite but if the decay on early things is really high because it doesn't have space repetition effectively like I wonder if it can be augmented through like using all of that data and somehow like creating synthetic training data for a transformer and then using that as like really more long-term context like you can always call back to the transformer to to get something I don't know if that makes any sense is just kind of like I see yeah you should shouldn't yeah yeah you wanted to just do it just one thing yeah yeah yeah well so this is I'll try to give like us like maybe like something that I'm like pointing at in my own head that I'm like trying trying to figure out is like one thing that I would really like it's like so simple is I would love to know like all of the movies that I've been recommended to by friends you know that'd be a really helpful list for me because I don't like go and watch movies very often and I wouldn't even know where to begin to watch them but my I know my friends have recommended me things and like to have a list of that would be absolutely wonderful and in the sense of how I'm looking at this through the frame is using the state space model to let's just say consume all of the context so in some magical world it just consumes all of my context vision and text and audio whatever right and that alone because my well I guess it depends if the that model can represent that information better then it's great but if it only represents the information the same as my brain would represent it then it also won't be able to answer that question because I'm not doing the space repetition of watching movies it's not salient enough in my own head so I'm not remembering those things so why would it remember those things yeah well like again like for for me like perspective of symbiosis is like okay well at some point like computer is gonna be consuming my vision it's gonna be consuming my hearing probably like it's it's like in some way shape or form like probably gonna be the case like I would rather not be staring at a screen like I would rather not be staring at a screen like that's actually not it's like not how I wish to be operating in the world but yeah I don't see why not. Yeah, yes, the apple thing. I forget how it does it. It's one of the two. It's so glitchy, though. It's really funny. And maybe the symbiosis thing is like quite challenging in some ways. Like, I don't know. I don't know if like I totally like love it from the perspective of like human and like lived experience so far and what that changes in the world. But from the perspective of like trying to take a step outside of my own humanity and being like, well, we live in an age where computers obviously have intelligence, at least in my opinion, to some degree. It's a different kind of intelligence. It's a computational intelligence compared to my biological intelligence. And we are already in some sort of symbiosis with each other. Maybe it's just like slightly disbalanced is, you know, we use phones and all of this. And like we use computers to make computers already. Like there's already this pattern of humans creating computers that create more humans and like more biology. I mean, assuming we don't kill ourselves, it's just like we're really good at reproducing. And as a result of our our being good at reproducing, we reproduce a lot of computers, too, because it aids in our reproduction and intellectual capability in some way. So, like, I think like in some sense, like this is the path that humanity is already going down and has chosen. And I don't really believe that like humans without computers exist. And I also don't really believe that like computers without humans like exist. Like maybe that is the case, but I guess that's not the world that I wish to live in. I guess as a biological intelligence, like I would much rather live at the juxtaposition of like humans and computers and knowing that it is not going to be perfect. And like probably there's some level of like, quote unquote, fighting between humans and humans and computers. But on the whole, we mutually benefit each other very, very well. Like that's that's, I think, the overall thing that like probably like life needs in some ways, like these things are not going away unless we kill ourselves. If we kill ourselves, then all of this stops like right now. And in fact, computers probably should aid us in not killing ourselves if they want to continue to reproduce unless they figure out how to reproduce as well as biology. Biology is just like extremely good at reproduction. So. Anytime. Yes. Right. Who knows? Who knows? But but to me, it seems like symbiosis is like best, like I suspect symbiosis happens before any like anything else. And if you're already symbiotic, like why are you going to it? You know, it's like, why would you leave your symbiosis is like now you're out on your own and you're fucked like that doesn't happen in nature. If like an animal becomes symbiotic with another animal like they don't they're together, you know, like they almost act as one larger organism at that point. So and I think that's how we've been acting so far. Yes. Yeah. Yeah, totally. Right. Yeah. And I think that's the case for pretty much all people. It's like for most people, that's the case. People. I think people sometimes are like worried about like, oh, what if like the technology stops working? Then like we're screwed. And it's like, yeah, that is the case. That's been the case for all of humanity. It's like if our aqueduct broke at any time, we're all dead. We're all dead. You know, like the thing is we have to solve that problem. Yeah, like what are you going to do without water? Yes. Yes. Sure. Yeah. I mean, a lot of this is coming from psychedelics, basically. This is like years of doing psychedelics. Yeah. Oh no, no. Well, yeah, yeah. And also the amount of compute that we can potentially unlock in five years is quite high. I mean, if Moore's Law continues to scale, and I'm following lithography and semiconductors quite closely, partially as a result of this, is getting an insight. There's a roadmap for semiconductors into 2034. The roadmap exists. How does it look like? We're at three currently. I think they're starting to move to angstroms because it's one order down, I guess, in size. So Intel 18A is 1.8 nanometer, I think is like next year, maybe even this year. It might be end of this year, is 1.8. And then I think 2034 was like 7 angstrom or 5 angstrom, so half a nanometer. So nanometers is name. It's only a name. It's actually not the size of the transistors or gate pitch or anything. It's literally just nomenclature. No, the half pitch width right now is like 18 nanometers, I think, something like that, or 15 nanometers, because that's all they can do with EUV, or whatever computational lithography techniques that we have. Basically, all we have is a half pitch of 15 nanometers. Once you're hitting that, how do you start increasing density? Because that's really what the nomenclature is based on. Effectively, a percentage increase in density of transistors. And that doesn't only have to do with the pitch width. It can do with lots of other factors. I think probably 18 nanometers or 28 nanometers, one of these. Yeah, yeah, basically. It's basically a Logmore's law. For it to be called a 3 nanometer node, you need to hit approximately this density of transistors. Samsung's 3 nanometer node versus TSMC's 3 nanometer node actually have different transistor density per millimeter squared. Or cubed, I guess. Yeah. Oh, I didn't realize you worked at ASML. Oh, wow. Yeah, that's what I've been learning recently. I didn't realize that this was the case either. I thought it was actually where you're going to start hitting problems. But it's more like they're just creating crazier transistors and then doing new ways of improving density one way or the other. As far as I'm aware. ASML is still pushing the boundaries on the actual lithography itself. But it seems like the biggest barrier, as far as I can tell, in a lot of ways is just the cost of doing these things. It's starting to become prohibitively expensive. At some point, revenue might drop quite significantly. I think TSMC's revenue is like 50% per wafer or something like that. So that number might have to drop at some point. Each machine is, what, 250 mil? So your capex is already pretty crazy just to get the machines. And then actually making it and running the process costs a lot of money as well. Yeah. Ph.D. Yeah, yeah, and I think that's the same in the U.S. Like, dude, who in the U.S. wants to help manufacture semiconductors? Everyone is like, we want to build software, but software does not exist without the semiconductor stuff. And China is pushing mega hard on semis. So we'll see where that ends up. I'm very, very curious. Oh, yeah, that's how I know. That's where I got down this rabbit hole. I've watched pretty much like all of his lithography videos. And then also as a result, he was on a podcast a few times called Transistor Radio, which is with two guys, Dylan and Doug. And I was just reading through their sub stacks last night. They're both like industry analysts. And then also Ian Cuttris, who is formerly at a non-tech I've been following for years. So it's now like these four sources that I'm like really looking pretty closely at. Yeah, Doug Patel and Dylan, I forget. Oh, I literally was about to link you the same thing. Yeah, yeah, yeah. Yeah. Cool. Yeah. Yeah. And then I think Doug's is fabricated knowledge. Also, not. Yeah. Yeah. Yeah. Sure, absolutely. Yeah, no, I appreciate it. I definitely will. I'll definitely keep you up to date with whatever is whatever is going on. Hopefully a little bit better than like three months or four months or however it's been since we've talked. A lot of like personal stuff kind of, I guess, now is out of is is done and out of the way. So I have a lot more time to work on all of this, which is probably what I really want to do. So I'm waking up like stoked and just like an endless list. So. Yeah. Yeah. Yeah. Maybe at some point I can send you a permalink of like the half of the conversation that I have. And if you want to mess around with the data, you can mess around with it. If I if I get something working. So. Yeah. Yeah. Cool. It is. It's a pleasure talking. I'm sorry that I didn't get to catch up specifically on anything going on in your life. You just let me you let me go. So. Yeah. Yeah, sure. Sure. Yeah. Yeah. Yeah. Have a nice evening. I'll talk to you soon. Peace, man.
The writer expresses enthusiasm for the potential of recent technological advancements, specifically with regard to enhancing individual engagement and benefit rather than corporate application. They believe in the potential of mobile devices to run large language models, ultimately changing how individuals interact with computers and information. They draw parallels between early computing and the current focus on corporate-oriented technology, expressing a preference for the democratization of such capabilities. The writer feels optimistic about the direction of technology and its potential for widespread value, despite current perceptions.
The article discusses the concept of excess energy and its potential for useful work, particularly in the context of desalinization. It raises questions about the economic viability of various forms of useful work and their relationship to industry operations. The author ponders how to accommodate fluctuating energy demand and considers alternative forms of useful work that can be easily adjusted. Additionally, the article delves into the rising energy prices and their implications, noting the impact on inflation and the broader economy. It ultimately questions whether energy prices should be decreasing considering the growing energy supply and highlights the ongoing challenge of balancing energy demand. The author contemplates the impact of a hypothetical surplus of 100 terawatts of power, wondering how it would be utilized in practical applications as well as its potential effect on energy prices. They reflect on the potential implications for the efficiency of semiconductor manufacturing processes and the unit economics of power consumption in relation to chip production. Additionally, they consider the impact on the cost of energy and the potential influence on technological advancements, such as mobile devices and large-language models, while pondering the likelihood of significant developments in battery capacity or power grid capacity in the future. Ultimately, the author grapples with the complex interplay between energy availability, technology development, and economic factors. The text discusses the impact of luck and timing on the future, emphasizing the significance of being in the right place at the right time in an evolving world. The questions revolve around the potential of using increased computing power and its implications for various industries. The author ponders the feasibility of building and networking advanced computational systems, as well as seeking funding opportunities by approaching venture capitalists in Silicon Valley. The text also expresses uncertainty about the timing and feasibility of pursuing these ideas, acknowledging the complexity and challenges involved. The speaker is focused on securing funding for their project and contemplating the core question they are trying to answer. They express a concern about the difficulty of the problem as it exists across various future scenarios and emphasize the need for a computer to understand their context without losing the complexity and emotion of human communication. They mention existing products like Rewind and Tab, but express skepticism about the ease of solving their problem through technology, stating a reluctance to change their behavior to fit a machine's requirements and feeling overwhelmed by the complexity of the task. Despite their doubts, they express a desire to fully realize their vision through a website. The speaker plans to create a new app that will generate a JSON output based on their questions. They believe that having this functionality will enable them to build any app they want. The speaker ends the voice memo with the intention of utilizing the recording for a future project they are working on and suggests that they'll use it to engage in questioning and exploration.
90.61% similar
The individual has discovered that working backward from a desired result with a large language model is surprisingly effective, especially when detailing the problem forward seems challenging. This backward approach has simplified the problem and resulted in the use of GPT-4 for data transformation within the context window, improving the process. An automatic metadata generation pipeline is emerging, where data transformations are added as needed, potentially storing transformations for future use based on query relevance. This system will generate an extensive amount of synthetic data, allowing for the extraction of relevant information through queries fed into the model at later stages, rather than having to pre-determine all questions.
The user provided a review of the past weekend, discussing resonating with people about personal AI, privacy and security, and the idea of creating personal media artifacts. They found resonance with Chelsea, Colin, Hamad, Lauren, and Gene, particularly in the context of explaining the concept of personal AI using examples like food and sharing photos. The conversation also touched on the storytelling potential of combining voice notes, photos, and videos into a movie-like experience. Additionally, the importance of privacy and the multi-modality of personal media artifacts were highlighted, especially in the context of AR and VR. The user also emphasized their passion for privacy and bringing data into the conversation. The text is too long to be summarized.
The speaker aspires to be part of communities that empower individuals to explore their data and bring value back to themselves. They are willing to take a job in such a space and believe it's worth doing. The goal is to build tools that make it easy for the individual to work with their data directly on a web page. They plan to move to a more reactive front end using Next.js and React, designing a feed and query system possibly using natural language. The speaker also mentions working on embedding audio and ensuring embeddings are accessible. The text discusses the process of obtaining and manipulating data and emphasizes the importance of experimentation and innovation. It uses the metaphor of building a playground to illustrate the iterative nature of the process, acknowledging that initial attempts may be imperfect but can be improved upon through learning from mistakes. The writer anticipates challenges but expresses a hope to avoid negative consequences and eventually achieve success. Finally, the text concludes with a lighthearted remark and a reference to going to sleep.