GRTiQ Podcast: 27 Payne

The following podcast is for informational purposes only. The contents of this podcast do not constitute tax, legal, or investment advice. Take responsibility for your own decisions, consult with the proper professionals, and do your own research.

Payne (00:22):

And this is the reason why I love The Graph so much because it’s so different than anything else. And it’s always, you always have something new to learn, you always find things to do, things to improve, things to work on.

Nick (01:06):

Welcome to the GRTiQ Podcast. Today I’m speaking with Payne, an Indexer at The Graph. Anyone who has ever visited The Graph Discord has already met Payne. He’s a very active voice and contributor to The Graph community.

(01:21):

In addition to his Indexer operation, Stake Squid, Payne has been instrumental at both The Graph Foundation and Edge & Node in various roles, including QA. I’ve had the fortune of having Payne on the podcast before as a panelist on the Subgraph Migration panel and on the Curation Launch panel. But this is the first time I get to speak with Payne one-on-one, and ask about his background and vision for The Graph.

(01:47):

Our conversation with Payne covers a lot of timely topics, including not only his ideas on the topic of keeping stake decentralized, but the mechanisms by which you would do it. We also talk about curation, his experience with the new Core Dev teams, and so much more. As always, we started the conversation talking about Payne’s professional and educational background.

Payne (02:13):

I started my university in Bucharest in Industrial Engineering. Did three years out of four, dropped out the fourth year, didn’t quite like it. And one of the reasons why I did that was because I found crypto and blockchain. And I’ve became very interested in that, and also got basically a job as a researcher in a small company that was focused on investing. So I continued with that for I think almost a year. And then I started a career in trading.

(02:51):

In the meantime, I was basically learning every single day. And yeah, I started a career in trading, been on my own basically for like another year and a half. And then I was just approached by a couple of friends of mine that were running CryptoCaliber, which is a media company basically researching writing reviews and research papers on crypto projects. And I’ve been with them for a while. And within that company I met Sebastian, and yeah, we basically started the Stake Squid. We both had some experience, him more than I do. I started learning as fast as I could.

Nick (03:37):

Where are you currently located?

Payne (03:39):

I’m currently living in Romania. I’ve been here my whole life. I’ve been born and raised here. Trying to travel a bit. But yeah, with all the coronavirus stuff, it wasn’t fun lately.

Nick (03:53):

What can you tell us about the people have Romania’s attitude towards crypto?

Payne (03:57):

We have a big community here. I think probably one of the reasons being inflation but it’s not that crazy as compared to like, let’s say Argentina or Brazil. But there’s a lot of people that have been introduced to crypto via Elrond, not sure if you know the project, but it’s a Layer 1. I don’t think that’s EVM compatible. But yeah, Elrond is a Romanian project that was … I think they started it in 2018. I was basically in one of their very first conferences. They were literally just bootstrapping that project and looking for investment back then. It was fun. So yeah, I think most of the people came in contact with crypto via them.

Nick (04:50):

So what motivates somebody who’s doing research in trading in crypto to become an Indexer?

Payne (04:56):

I know I’ve been all over the place which is weird, but it really helped because especially on research side, because I knew exactly what The Graph was doing, I knew exactly what it was useful for and why people would actually use something like this.

Nick (05:12):

You and your co-founder at Stake Squid, how did you first get involved in noncustodial staking services?

Payne (05:21):

Yeah. His name is Sebastian. He’s been a computer scientist for most of his life. And it was running stuff by his own because he didn’t have, I think anyone to be on his side. And yeah, it was just, it was learning, playing around, running a bunch of validators, I forgot which nodes. He also ran miners. So he definitely had tons of experience. And I saw him doing it. He was talking in our internal chat. And I was like, “Hey, can I do this with you?” And then yeah, basically, that was it, that was the story.

Nick (06:02):

Do you remember when you first became aware of The Graph and what drew you to the project?

Payne (06:06):

Yeah. We found out about The Graph a couple of months after the company was formed. We couldn’t engage in the community or in the stuff that they were building. But eventually, Sebastian found a blog post that summer in July that basically opened up the testnet for Indexers. And we also like, back in the day, we had a bunch of experience that we thought it was enough for us to jump in. It was definitely not. We definitely learned so much, so much new stuff.

(06:47):

But yeah, we jumped in. We had a blast. I was very intrigued and hooked up by the community and by the project and by what everyone was doing. So I kept grinding and learning new stuff and helping other people understand the network, and understand how the different roles of the protocol worked. And yeah, that’s kind of the story.

Nick (07:14):

How did you get involved with The Graph team?

Payne (07:18):

I think it was November last year, or late October. I had a DM from Yaniv Tal. He said like, “Hey, do you have time to jump on a call?” I said, “Yeah, why not?” I never spoke with him before, but I was always keen on learning or speaking more with the founders because I was interacting on kind of daily/weekly basis with a Jannis in the chats of Discord during the testnet. So yeah, I jumped on a call with Yaniv. And yeah, we chatted for like half an hour. And then a couple of weeks later, another chat. And then I had my contract.

(08:00):

So if you want to see exactly how those conversations evolved, I highly recommend watching the EthCC recording of Yaniv’s panel. It was Yaniv and Eva Beylin talking about these things in Paris. It’s a very, very interesting panel. I highly recommend it.

Nick (08:27):

So you work on multiple networks with Stake Squid, not just The Graph, here at Solana, Avalanche and others. What do you do across these different networks?

Payne (08:39):

So yeah, we run validators for I believe nine different networks or protocols. We’re currently in the works of adding two more in our portfolio. But mostly me personally, I’m mostly focused on Graph, Solana and Agoric. Sebastian has the rest.

Nick (09:01):

How is The Graph different from some of these other networks you guys are working on?

Payne (09:04):

Oh, man, don’t get me started. Everything else is quite literally at this point, given that those networks are using principles quite old on the market. Stuff like Solana for example is literally just you plug a few commands and you press Start and it magically works.

(09:22):

For The Graph we try to do something similar by running, like by having our Docker compose stack that I think most of the network uses right now, but it’s a lot more different … I don’t know how to explain it. The Graph has so many moving parts compared to like Solana for example. In The Graph stack, you have The Graph Node, which you can split it into two, the query and the index node. And then you have the index agent, index service, the database or databases depending on how much you want to scale or have redundancy and stuff. This is a very complex system compared to Mina, Solana, Avalanche and that sort of stuff.

(10:11):

And this is the reason why I love The Graph so much, because it’s so different than anything else. And it’s always, you always have something new to learn. You always find things to do, things to improve, things to work on.

(10:28):

I’m not aware if you know, but recently I’ve been working on the testnet repository. I had an advanced configuration that was using as a separate branch, but now I integrated it into the main branch, and technically speaking, anyone that was running the old version before, and upgrades now, it’d be basically seamless upgrade, you don’t need to do anything manually, just make sure that you have the right environment variables, which is fun. And it’s a lot more configurable than the normal stack that we were running with, for example.

(11:05):

We were planning to work on that going forward to make it even more robust and to give people even more choices of configuration.

Nick (11:14):

So for listeners that wouldn’t be familiar with this idea of testnet repository, what is that?

Payne (11:19):

It’s basically a GitHub page where you can view instructions and download all the necessary software to basically spin up your graph node.

Nick (11:29):

So how is The Graph’s community different from some of the other protocols that Stake Squid works with?

Payne (11:34):

This is a very good question. Well, in the case of The Graph, we have people from all the other networks that are contributing to the ecosystem, that are building subgraphs. They are running their own nodes. And this is one part of the community that’s very different. Because like, yeah, I mean, on Solana, yeah, you have people building on Solana alone, but on The Graph, you have people building on BSC, you have people building on Polygon. Yeah. And it’s a much more diverse community.

Nick (12:07):

You mentioned earlier that there’s a little bit more complexity in working at The Graph than some of these other protocols. Can you explain what that complexity is and how it’s different?

Payne (12:16):

Yeah. The Graph has so many moving parts. It has to make sure that everything runs well together, and nothing breaks whenever there’s an update, right? And the fact that there’s also a disparity between the hosted service and the network, right now is also one of the complex parts, because you have to make sure that whenever you upgrade The Graph node, that doesn’t break neither of the networks. I know, eventually, the hosting service will eventually be deprecated. But until there’s feature parity, I think we’re still going to have to deal with this struggle.

(12:57):

The Graph is a very complex system that needs to be very carefully designed and graded so that nothing breaks. From the perspective of a validator, again, it’s a whole architecture that you need to design and run compared to Solana which is just a single program, basically.

Nick (13:17):

Payne, one thing that makes you unique is the number of Indexers you run at The Graph. At the time of this recording, I believe its five Indexers. What can you tell me about running that many Indexers at The Graph?

Payne (13:29):

It’s a bit tricky, because one of the reasons why we are running this many is that we couldn’t combine the own stake from different vesting contracts into a single one. So we’re forced basically to run multiple Indexers. And this affects our profitability because each Indexer will have its own costs associated to too many transactions on the network. So the more indexes that you run, the more you have to pay in operational costs per month. And it’s a tricky situation. You have to make sure that you stay profitable if you’re going to add more Indexers on-chain.

Nick (14:14):

Well, in addition to operating that many Indexers and being very involved in The Graph community, you also work alongside The Graph Foundation. You’ve been featured on this podcast before as a panelist when The Graph launched its curation program. And then early on, we had a panel about when we talked about the migration of subgraphs from hosted service to the main net. How would you describe your role or relationship with The Graph Foundation?

Payne (14:38):

Right. My work is split between working with the foundation and working with Edge & Node. I don’t quite differentiate that much in between them. Like on the foundation I don’t really have a quite clear role, but then on Edge & Node I do help with QA. So I’m all over the place. I don’t really have one single thing that I’m doing. I’m doing lots of things whenever I’m needed basically. I jump on calls with the docs team. I jump on calls with designers. Yeah, I’m all over the place. I’m not really having one single role.

Nick (15:18):

Why are you so involved in The Graph and wear so many different hats?

Payne (15:22):

Well, I think one of the reasons why I’m so involved, how I got into this spot is that I, immediately once I joined the testnet last year in July, was like I was hooked up by the tech, by the community, by everything. So I just spend 12 to 18 hours a day, on average, basically just learning and being active and experimenting and doing all sorts of crazy things in the community and in the network. So that I have accumulated some, I believe, I could say, the vast experience. There’s not too many people that know as much as I do without trying to brag too much. It’s not even a brag, but it’s just I wish that it were many more people than with the knowledge that I have, in a way. I don’t know how to say.

(16:25):

So yeah. If you think you have the knowledge to join the Edge & Node team, yeah, I’m just letting you know that we’re hiring a QA and Support Engineer on the US, I believe. So yeah. If you want to apply and you need a link, I can send to you in DMs and over Discord.

Nick (18:32):

What is the role of QA? I know you’d mentioned that in our earlier panel we did with Nena and Juan discussing the launch of Curation. What does the QA role do?

Payne (18:43):

Well, making sure that the product works. That involves all the features and all the components of the explorer, as well as the studio. So just making sure that we don’t have any bugs. If you find some, report them, make sure they’re fixed. Recheck after they’re supposedly fixed, just to make sure that they are indeed fixed. And yeah, make sure that once those fixes make it to production to recheck and make sure that they are again, indeed fixed.

(19:21):

So yeah, you need to have some kind of knowledge of the protocols so you can find edge cases abuse the system and find problems before other people find them and get stuck. For example, I don’t know, delegation transactions not working or something just saying it as an example.

Nick (19:39):

As I mentioned, Payne, you joined me for a podcast we did a while ago. When Curation services went live at The Graph, you, Nena and Juan joined me to talk about that important event. I want to kind of switch perspectives now and put you in the role of an Indexer. As we’re now a month or more past the launch of Curation services, what’s your perspective of Curation as an Indexer?

Payne (20:05):

Man, it was chaotic in the beginning because everyone was seeing new subgraphs, everyone was signaling them and a couple of hours later un-signaling and shuffling GRT around the protocol, which created basically a whole lot of chaos. And when I’m saying chaos is that, the reason being is that the Indexer rewards are distributed to the whole network, proportional to each subgraphs signal value on the network. And whenever you have subgraphs having 100k signal now, 200k later, and 10k in a day, you’ll get people, Indexers to allocate to set subgraphs, and basically be stuck on those subgraphs until they sync those subgraphs up the chain head so they can close the allocations.

(21:06):

This basically, this whole chaos was that people were seeing subgraphs that had lots of signal, they were allocating to them. And then hours later, they had zero signal because people shuffled those GRTs into other subgraphs. And yeah, I definitely learned my lesson. I was forced to close, I think six or seven subgraphs with a so-called 0x0 Proof of Indexing. So I gave up my rewards. But that was only for, I don’t know, less than 12 hours of waiting that between the moment that I allocated and when I closed the allocation.

(21:49):

But in the meantime, things have been stabilized a little bit. There’s not that much chaotic. Most of the signal has been settled on promising subgraphs, for example, Sushi, and the migration partners. And the EIP-721 and the other one, I forgot the name of it, which are legit subgraphs. And yeah, things are looking great again, now. Hopefully it will not be so volatile, again, because it’s very time-consuming and very frustrating both as an Indexer and probably as a Curator as well.

Nick (22:29):

What’s your advice for any listener that’s either thinking about or currently involved in curation, but just still learning it?

Payne (22:36):

Curation is very, very hard, because you have to both assess the probability of that subgraphs to be used in a future and how much is that subgraph going to be used in the future, because 10% of the entire query fees that that subgraph generates will go to Curators, but if that subgraph won’t have that many query fees, you won’t earn anything. And also, the earlier you are on the subgraph, the more shares you can grab. So that’s also, you have to think fast and risk getting rock bold, or stay on the sidelines and eventually see that that subgraph will actually do something and you missed the train.

(23:27):

From this perspective is very complicated. And yeah, if you want a more passive role without so much risk as a Curator, it will probably be better to just settle to be the Delegator instead in my opinion.

Nick (23:48):

One concept I really haven’t understood is when Curation launched and you referenced this just a minute ago, there were some, I don’t know, fake is the right word, but there were some non-legit subgraphs. I don’t understand how that happened or why that happens. Are you able to explain that?

Payne (24:04):

Yeah, sure. So the network is permissionless. That means anyone can take any actions in the protocol without anyone blocking you from doing so. Right? Some people so that they can basically do that. And they forked legit subgraphs with legit code, and they published them on the network on their address and immediately signaling them. So anyone that was coming after said people would basically get a lot less shares so those people that basically published the subgraphs could get out on a profit. Yeah, like I said, those subgraphs, the code of them, like the index, data of their subgraphs, was legit because most of them in the beginning were actual clones, forks of real production subgraphs. But obviously those weren’t deployed by the teams behind the subgraphs, or who developed those subgraphs. There were just a bunch of kids that tried to make some GRT on the network. Yeah.

Nick (25:19):

So to better understand this, if I used The Graph as a user, I have a dapp that I pull data and I write a subgraph. If someone forks that subgraph and deploys it and starts sending signal on it, does that mean I have to use theirs now, or can I still use the original one I developed? What’s the friction between the two?

Payne (25:42):

That’s a very good question. Yeah. The way it works is that the code itself, when you compile it, it gets written into an IPFS hash. That’s a hash of the code of the exact code, all the letters in the subgraph YAML file, all the letters in the mappings file and stuff like that, all gets compiled into a single line of 24 characters or something like that. And the idea is that if you don’t want to use someone else’s code, you just modify one letter and you have a totally new subgraph, which hasn’t been heard of in the network. So you just publish that, signal on it, and it’s yours.

(26:31):

If you do that, you also have control over how you upgrade that subgraph. In the event that you need more functions in it, or if that subgraph fails, it’s only the creator of the subgraph that is able to upgrade or modify the data inside that subgraph.

Nick (26:50):

Payne, I really like your advice for listeners who might not have the comfort level or the knowledge to participate in The Graph as a Curator that they can still participate in the form of being a Delegator? If you were to give advice to listeners that are new to The Graph, or are new to delegating, what would your advice be for them?

Payne (27:10):

First of all, APY doesn’t mean anything if your Indexer doesn’t know what it’s doing. Second of all, decentralization is very important. So you should be scrolling down more than just the first 10 people on the network. Third of all, be on the lookout for people that are active in the community and actively helping people out, that are participating in weekly discussions because these are the people that you can know for a fact that they know what they’re doing and they will perform great.

Nick (27:47):

Decentralization’s certainly been a hot topic within The Graph community recently. There was a post in the forum by Oliver about some ideas, starting some conversation about keeping stake decentralized. I had Chris Remus on the podcast, also an Indexer at The Graph, who has a great emphasis on keeping stake decentralized. What’s your opinion on this topic?

Payne (28:09):

Yeah. I mean, decentralization is one of the most important aspects of the blockchain. This is why crypto was created in the first place. So we should be aware of certain patterns at the core and many other different protocols and try to learn from that, and try to protect the network in a way that it doesn’t become centralized. Because if you only have 10 or 15 Indexers that control 80% of the network, then you don’t have anything more than just another hosted service just run by a bunch of randos all over the globe. 10 or 15 randos. That is not just the single server, but that’s still not decentralization, right? Because half of them started working together with malicious intent. They can seriously impact not just a bunch of people in the protocol, but whole applications that rely on set data. And this is not something that we want to have happening in the network, right?

Nick (29:53):

Payne, how would you address the issue of decentralization at The Graph? I mean, I understand it’s an important topic. It’s certainly at the top of a lot of people’s minds right now. What are the mechanisms that you would want to see implemented to ensure that stake is kept decentralized?

Payne (30:09):

Right. I think first of all, you should think … you should put yourself in the boots of a regular Joe that doesn’t know or doesn’t care about decentralization, and just opens the page and sees big numbers and clicks a button on the biggest number, right? Because big number good. So I think, first of all, we should address the way that data is displayed on the Explorer. Right now is I think it’s ordered by self stake. So if you look on, if you open The Graph Explorer, you will see basically that, I don’t know, the first 10 or 15 Indexers have control over like 80% or 70% of the delegation market right now. So that’s one of the first things that I would do.

(31:05):

Is definitely not enough, or the people that are currently delegating won’t re-delegate just because we changed the number, the order number in the UI, but at least it helps future Delegators that want to take a decision. In the same forum post, I think Oliver pointed out a very nice little detail. Solana Beach, for example, has the first X number of validators hidden in a way that you have to click a button to see them. And you will see the rest of the people where you can delegate without contributing to further centralization of network, which is one of the best ideas that I’ve seen so far, in the way that people display data of the network, which is really nice to see.

(32:03):

Other things that I would do. I think Chris had a great idea, which I also had a while ago, which was having some sort of dynamic delegation cap. Right now everyone in the network is limited to 16 times their own stake, basically, so that people that self-stake the 100 million GRT, can basically eat up a quarter of the entire supply of the network in their own delegation, which is really concerning. You only need four of those people to basically have the whole network centralized onto four entities, if everyone delegates to them.

(32:51):

And one of the ideas was to have this 16 times ratio dynamic based on how much own stake you have. The more you have, the less delegation capacity you’ll have. So that would eventually limit in a way decentralization, one of the centralization pain points basically. I hope. We’ll see. We’ll see if that even gets implemented, but I hope it does.

Nick (33:20):

I think it’s important Payne to emphasize the responsibility Delegators have when it comes to helping keep stake decentralized. How would you help Delegators think through their role in this important issue?

Payne (33:33):

Yeah. I think Delegators should keep in mind that having a lot of self-stake in the network doesn’t always mean that those Indexers actually know what they’re doing. I don’t want to call names. But yeah, this rule is definitely applying in the current time. The idea is that you are contributing to the network more than you think so because Delegators have strength in numbers. There aren’t so many people that know how to run Indexers, so they will be Delegators instead.

(34:17):

Yeah, even though we do want a lot of Indexers on the network, we won’t have as many Indexers as Delegators in the network, which is something to keep in mind. We need Delegators to be aware of the centralization risks and take action in that way.

Nick (34:36):

At the time of this recording, there’s I believe 159, 160 Indexers at The Graph. As you think about the future of The Graph and the Indexer community, how much more do you think grows? I mean, how big does that number get?

Payne (34:51):

Well, we have 7 billion GRT that is not delegated or used in the network right now. So there’s definitely room to grow. The more Indexers, the better. The more Delegators, the better. The more Curators, the better. Because especially on the Indexer side, the more Indexers we have, the more redundant the whole network is. If one goes down, we have 500 more to take his place, right?

Nick (35:22):

What’s been your experience, both as someone that is very technically involved in The Graph, but also running Indexers at The Graph, what’s been your experience with the new Core Development teams with StreamingFast and Figment?

Payne (35:34):

So I can talk about StreamingFast because I had tons of interactions with them and with their funders. They’re really great guys. They’re building something that will really, really revolutionize the way that Graph works and will work in the future. And I’m super excited to get their technology properly implemented into the network. I don’t know how to explain in simple terms, but it’s a difference in night and day in the way it works compared to the way The Graph Node works today. And we need to find basically a middle ground in the way that these changes get implemented in The Graph Node, so that it’s compatible with the current subgraphs on the network and it’s also compatible with the network itself, and how the Indexers are running it. It’s a very complex matter. It’ll be a very nice year coming to us and watching those development integrated in the currently working product.

(36:48):

And I also wanted to point out that StreamingFast has recently open sourced their software. We’re part of it. They’re in the process of open sourcing everything. And I highly encourage if we have any Indexers listening to the podcast to give it a shot and see how it works, because this is the future. And we better be prepared to learn how things work so that we don’t have any surprises when it comes later and we will be needed to run it.

(37:26):

Same as with Erigon nodes, Erigon being an Ethereum client that we’re currently trying to find out if there are any bugs and things that needs to be fixed.

Nick (37:38):

Payne, I want to ask you because of your technical background and involvement with The Graph, as well as Stake Squid’s involvement with a lot of different networks. What do you think is the long term potential or impact of The Graph?

Payne (37:52):

Well, one of the reasons why we’re so happy to be involved in The Graph is that it’s always such a dynamic environment with potentially so many chains that someone would interact with. I think Graph’s future is very bright simply because if you think of how many people are using it today, just imagine expanding to all the other chains that exists, for example, Polkadot, NEAR, Solana and so on having all the other developers running things on The Graph, or with the help of The Graph.

(38:28):

So my vision for The Graph is that is going to be amazing and it’s going to be used by everyone else. And the vision of Stake Squid with is that we need to be part of the rest of the networks to learn how to operate them when it comes the time to run those nodes for The Graph as archive nodes.

Nick (38:49):

If listeners want to learn more about you or the things you’re doing at Stake Squid, what’s the best way to stay in touch?

Payne (38:54):

Hit me up on Discord. I’m there 12 hours a day at least. I always do my best to answer DMs. If not, if I don’t answer your DM, just write again. I will eventually. I’m trying my best to answer within the same hour when I got the DMs, but sometimes I’m clicking the icons and with the thinking that I will reply in a bit and then I forget, so I’m very sorry if that happened to anyone. Just hit me up again and I’ll reach back.