Podcast audio transcript

[0:04] Welcome back to the drupal Easy podcast. This is season 17 episode number three. In today's episode, we'll be talking with Ryan Price about a project. He worked on to update a drupal Seven site with extensive backend integration to drupal 10, including some of the challenges they faced and the solutions they used before we get to the interview. Allow me to remind you that upgrading drupal seven sites is one of the core businesses here at drupal Easy. Mostly thanks to our extensive experience with data migration, contact us for more information. Welcome back to the drupal Lucy podcast. Mr Ryan Price. How are you doing? Super well. It is summertime in Oregon and things are happening. The fish are jumping the cotton is high. None of those things are true. Let me ask a question. I just gave a presentation today where I had to give people my background in drupal. And a couple years ago, I stopped counting. Like how long I've been in drupal and I just changed it to I've been in drupal since 2006.

[1:15] How did you answer that question? How, how long have you been hanging around the drupal community. That's a good question. I, I guess I just say usually that I've used more versions of drupal than most people own cars, something like that. Well, that's not too difficult these days with the rapid pace of, well, major, major updates. So, but even, even before, right, I still, I still have had, you know, more, more drupal than cars, even if you only got to 80, so, actually you're talking about ever, like, cars ever owned? Not currently owned, but how many cars you've ever owned? Yeah. Yeah. All right. Yeah. So you and I have been around for the, roughly the same amount of time. Very, very similar time spans. Yeah. So, what are you doing these days? Who are you working for? I work at a consultancy called IC F, it is a really big company that you may not have heard of unless you work in government. I don't think it's exclusively government, but there's a lot of that in there.

[2:16] And I'm working on a government contract so it's, the health and human services is a department of the US government and then there's sort of like a bunch of sub departments under there. So I'm going to be eventually in charge of a whole bunch of health and human services sites. As long as they keep our contract up there, it's government stuff is its own special beast that I am learning a lot about. I've done some things with like municipal governments or I did one with the state government in the past. But the, even at the federal level, it gets more layers of complexity. So if you were meeting a Dr another drupal developer for the first time, and they asked, what do you do with drupal or what's your drupal background? And you know, you're at a party. So, you know, it's gonna be a quick conversation type of thing like, well, how would you answer that? Yeah, I started doing drupal in very early days. Drupal 4.6 was the first drupal I ever used and I started helping out at a meet up and um some drupal camps locally in Florida where I was living. I started working on this podcast and some training exercises. I've done session selections with the drupal Association for the drupal Cons. I've done mentoring programs with drupal Association. I actually helped start something that was like a precursor to that.

[3:42] And I've done a whole bunch of websites including things like pfizer or the Olympics for NBC Colleges and nonprofits and now federal government websites. So, and you're kind of a jock of all trades. Right. I do a lot of everything at my current job. I actually do a lot more Linux system administration than. Anybody has ever asked me to do professionally. It's something that I always kind of did on the side because I was doing like self self hosting of my own site and things like that. And I never really wanted to pay someone, you know, a big pile of money just to give me the really basic thing that I could do on my own. So I've never had my personal site on like a, you know, platform as a service type of a host, but plenty of client sites I do. But where we are now, we actually have a lot more of that. Like we're given a virtual host and we have to set it up ourselves for reasons. And you are currently involved in helping to organize an event that's coming up in the drupal community. Correct? Yeah. Yeah. So in the Pacific Northwest, a lot of the cities are pretty closely located.

[4:52] Uh Well, for the west coast close, it takes me like three hours to get to Seattle and maybe six or seven hours to get to Vancouver. So between those three cities, we have a rotating camp schedule every, every other year, it moves to a new city. So this year is going to be Seattle. It's the first time that we've done drupal camp since before the pandemic. And it's called the Pacific Northwest drupal Summit. It's gonna be in Seattle October 11th through the 13th. And we're going to have a lot of sessions, day and a half of sessions. We're going to have a contribution room. We're going to have training by somebody that you might know Mike is going to come and do his uh professional, what do you call it? Professional development tools? There you go, professional development tools, workshop on Friday. And there will be a social time and getting to hang out in downtown Seattle and particularly around the University of Washington campus. It's a really cool space and we're going to have the whole building to ourselves more or less for the days of the camp. So it should be a really great time and it's not expensive and you should come check us out. And we will put a link in the show notes, super duper. All right. So we are gonna talk about a couple of projects or maybe it's one project. I'm gonna let you explain that in a minute that you've been working on lately. And the super short version is these are dr these were drupal seven sites with a lot of back end integration.

[6:21] That have moved to drupal 10 sites where that backend integration has been completely redone using modern methods. So we're gonna kind of talk about that process a little bit. So why don't you tell us about the, the, the project a little bit? Sure. So I will say this is a project I did for a former employer and I'm going to decide not to mention their name if I can help it during this episode. But it's not like it's a secret. I worked on the site um for over a year and between the 22 sort of like projects that were sister together as it were these two organizations, they have study abroad programs. So what you do is I'm a college student and I would like to go travel to Spain or to Germany or to Japan. And if you're in the United States, you deal with IES. If you're in the Asia Pacific area, you would deal with a company that's called the Study Abroad Foundation or Safie si think used to have an acronym that it meant something. And now it's just like a lot of modern company names. It just, it just IES stands for IES. So it's IES abroad.org. If you want to look it up, they have a couple of 1000 students that they send abroad every year just from the US to various other parts of the world. And they actually have.

[7:49] A dedicated building at a lot of their partner schools. So you will go, you will stay in the same dorm room as all of the previous study abroad students and your classes are actually taught in a space that's more or less dedicated to the study abroad program. So they have an interesting set up where you're not just like signing up to go to another school. You're, you're going to have this whole like kind of catered experience. So along with that, there's the admissions process and if you've ever applied to a college before, or, you know, apply to a government job, you know, that there's a lot of paperwork that goes along with it, you have to prove, you know, who you are, give a lot of information about yourself, traveling abroad. They're, they're also trying to help this experience be as easy as it possibly can be for all the students, the parents, the schools who have to, you know, grant credit to these students afterwards. Everybody involved. There's, there's lots of people and lots of moving parts. So it's, it's not an uncomplicated thing. Like, hey, I want to buy tickets to go to a concert which is still plenty complicated. But imagine having to buy tickets to a concert that lasts for three months or six months. So, there's a lot more details that you need to collect. So, what was your starting point?

[9:11] Yeah, they had a drupal Seven site that was built, you know, in, in the pretty early drupal seven era and lots and lots of custom code. There were some files that had thousands of lines and dozens of functions in them.

[9:27] Not, not any like really terrible things like you would see like sequel queries and the templates or anything like that. It was pretty well done for what it was and, and given that it was organically grown over time as, as you will with any project that lasts for seven or 10 or more years, you have, you know, you'd always have to make that trade off between. We need to finish this new feature now versus we need to go refactor everything so that it's easy to maintain later in talking to you about this before. Uh we recorded, you mentioned that the, the process is actually managed both on the drupal site and also in a back end application, and both those things need to talk to the same data and pass data around. Exactly. So is that, is it safe to assume that, That's still the case with the drupal 10 site. Yeah. Yeah. So this, this is kind of the biggest challenge with this project. There are a lot of things we're going to kind of focus on just this back end integration. When for example, a student, you know, wants to sign up to apply to the program, they need to get a record in this back end database. When somebody inside of the IES organization wants to review that information, they wanna review that information, but they need to be able to see more things. So they have their own in house, homegrown things, but only IE employees can use that and see that and it sort of like exist inside the firewall.

[10:57] But then the person from the school who is like your college advisor, uh sometimes there's a person dedicated to study abroad. Sometimes it's somebody in the admissions office, it kind of depends can go into that interface and approve students who want to go on these programs. But in order to do that, they don't have access to the IES firewall. So we need to give them a web application. In this case, it was a React application so that they could view that stuff. It was, didn't used to be a React application used to be drupal seven. So there's, there's a lot of moving parts.

[11:32] Um, a lot of people need to touch all the same records over the course of could be 3 to 4 weeks. It could be 3 to 4 months. It kind of depends on timing and everything like that. And then once you kind of get accepted and you're in the program, the drupal site doesn't really matter anymore. This really is a conversion website to get you from being I don't know what IES is. I know what IES is. Now, I need to decide where I want to go. Once you decide where you want to go, I need to apply and then there's a whole several step process of application and, and tracking that was really kind of the the big meat of this project. So the way I, I think that this would be approached as one of the early decisions has to be, where's the canonical source of data? Is the drupal database? The canonical source of data is this back end system, the canonical source of data? Are we constantly syncing data between the two or is it only synced when certain actions happen on one side or the other?

[12:34] So I guess my main my question is as far as all those decisions, did you just match what was happening in drupal seven or was there kind of a, all right. Let's step back. Let's look at the things that really worked in drupal Seven and, you know, versus things that didn't work and refactor and fix some maybe long standing issues in drupal 10. So, what was that thought process like? Exactly. Exactly. That, so in drupal seven, a lot of the records were created first in drupal seven synced to the other database. But because of the way the code was written, it needed to be. Perfectly in sync and we needed to have a copy in both places. And myself and John Kba, who was another tech lead that was involved with this project. We both looked at that and we said we wanna do better. We also decided to rethink the information architecture, the the sort of like content types and things they would use to represent all the data on the drupal site to store as little as possible on the drupal site as we could.

[13:47] There are certain things that just because of when they get collected. So the first time that you fill out your application, all of your like personally identifiable information, all the, all the sort of information from your application needs to be stored on the drupal site just so that we can access it very quickly. And there's, you know, like really short round trips, right? Once you say submit the application that now becomes locked, it's read only essentially. And then we'd now start treating the back end database as the source of truth. And from then on, yeah, go ahead when you say submit the application. So what I, what I'm envisioning is that the person actually filling out the application that might be done in more than one sitting. Exactly. They might sit down, start the application, get a third of the way through and say, oh, I need to go get this document and this document, let me save the application where it is, come back a day later, add more info and it's only when that document, that application is complete. So they hit the button and as you said, it actually submits it, it locks it and then the next the next step of the process, right? And if you think about the time before the drupal seven site.

[14:58] This was a paper application and so submit was put a stamp on it and put it in the mail, right? So that submit step is similar to put a stamp on it and put it in the mail. And then from then on, I guess they would have to call you or send you back a letter, you know, in order to make changes or updates. But you can kind of think of that process of like sending something in the mailbox is the same thing for us as sending it to the back end database. So were you using a content type or a custom entity or a web form or something else as the actual drupal? I'll use the word entity here to collect that data for the application. We ended up deciding on web form. And I think the number one reason why I picked that was because I knew that sometimes the questions have to get changed. So maybe like they want to rephrase the wording of something. Or um one of the things that we added as we upgraded to the new site was they added a whole bunch of more fields around. Your gender representation and your pronouns, that was a new field that they were adding to their application that they didn't have in the drupal seven days. So we knew that they were going to be adding new questions. We knew they were going to be updating the language on things. So we said, well, web form is a lot easier to maintain than a completely custom thing. Or we were thinking about doing something with javascript at some point. But other reasons, the timing just didn't really work out on making that happen.

[16:27] And we looked at doing something like a content type and we ended up going with entity with um web form. And I think doing it again, I probably would have done something a little bit more stand alone that didn't care about drupal as much. Another big advantage of doing it with web form is you could technically start filling it out anonymously. And web form has a really great mechanism for transferring something from anonymous to not anonymous. But part way through the project, we decided you, you know what, you're just always going to have to be signed in. So that anonymous bit of the form was actually a custom form anyway. So at the end of the day, I think I would have done it a little bit differently, but it was, it was already mostly finished by the time we got to that level. So, so from the student perspective, they go through this process, they submit the application, they, they hit submit, they're not able to access their application anymore. Maybe it's only in read only mode, it goes to the back end. So is the student still using the website for updates on their application or what does that look like? And um it sounds like that something's happening on the back end, but then data gets sent back to drupal in some form. So maybe talk about that. This is something we spend a lot of our time and effort on because you know, the the business reason like why are we building this website is number one.

[17:52] You know, make the application process as simple and easy to do as possible.

[17:58] You know, do do as much as we can to make that application easy to use. You know, there's lots of work done on the language and the colors and the interface and making it, you know, nice and, and modern. And then once you submit your application, now you are in this state where say I have started my application and there's a dashboard.

[18:19] Technically, you could get to the dashboard before. But the dashboard for somebody who has not finished their application and somebody who's just finished their application were about the same. Um You know, with the exception of there's a special widget on the page that says your application is not finished yet, please hit this button to finish the application, right? So we have a custom content type with a lot of very highly contextual blocks on them. And that piece of content actually had seven different, states which were basically represented by seven different nodes and 11 node could say, hey, this is the, I haven't finished my application stage. This is the I have finished my application stage. This is the IES has approved my application. This is the my school has approved my application. This is the I need to fill out some more documents stage. This is the I'm about to go on my trip stage and this is my trip is over stage, right?

[19:21] And because each one of those had a different layout, they wanted to feature different information like move it up higher in the stack or lower in the stack, they wanted to show off blog posts from other students who have also gone to England where you're going to be going at the beginning. If you don't know where you're going, we don't have the ability to feature some of those, you know, program specific things or the phone number of the person that you can call at your center. Once you get to your location to get more information, things like that were not available, there's also a really big packet of all the information you didn't know about going on your trip. Like, you know, how do I get a phone card once I get there where it's the location of the US embassy in that country. All this information that you can download, they don't give you that packet until. You're basically done with the process because there's a lot of program specific stuff in there that actually changes from like, even which quote unquote, which major are you taking? Right. Are you doing the, the cultural studies? Are you're doing the language study? You're doing the, you know, the music program or whatever thing it was, they would all get a slightly different version of that packet. So lots of the highly contextual things that show up on these dashboards.

[20:33] Some of it's driven by your user profile and saying I'm going on the spring 2025 trip to England and my program is the, you know, government program, whatever thing that you're there to sort of like cos study while you're on your trip, we store all that stuff in your user profile so that we know when we want to present one of those contextual blocks. What do we show you? So you're not using the workflow? Is that the module that shows you different workflow states? Yep, it sounds like you're not using that. We were, we weren't using that. It was something very, very similar to that and even to where there are certain states, if you're trying to go from state A to state B, you know, workflow has some logic built in so that you can only kind of go from one to the other. But one of the reasons being there's no button on drupal that anyone can push that will make that state change. The only thing that can make that state change is the back end data changing, right? So a lot of those activities are dependent on outside factors. And I feel like a big advantage of using something like workflow is it's got the U I bits of it built in in addition to the data structure.

[21:45] So even if, yeah, I mean, again, like that, that definitely could have been a choice there because once you look at the gymnastics that you have to go through to put someone through all those different states, there's a lot in common with the workflow states. Yeah. So how did you send data from the back end? You know, when an event happened on the back end that had to change something on the drupal side? Was that just you create an end point and had an end point or how did that work? Yeah. So when drupal wants to send something to the back end, we put together a, you know, big PHP object, we serialize it into JSON and we send it to the right place. You know, we have to have our API keys set up correctly and things like that direction. And yeah, for that, it was pretty, pretty similar stuff like, you know, make, make a curl request, get access to the Json UN serialize it and, and deal with it as we needed to.

[22:47] So, other than web form, were there any drupal contrib modules that were like vital to, to this integration? Yeah, because the interesting thing is, you know, we're, we're doing the pushing and pulling and I feel like a lot of the contrib modules that I would want is if drupal ended up being the source of data. But in this case, drupal was like a client, right? So a lot of the stuff we kind of had to do a little bit more custom. Um We definitely did a lot of good, you know, like of the underlying API S, you know, the, the drupal libraries and the symphony libraries and things, those, those got used a bunch. I was, I will say we did actually have to do some things that, that felt like we were working with those drupal API S when we did the.

[23:33] There's a front end that exists for the people from the college, right? The college employees and I mentioned this very briefly because they're not inside the IES firewall. They actually had a a it's the same drupal site, the same login, but because of their user role would send them off to a different dashboard. And that dashboard said these are the students that belong to my college. These are the ones who are, you know, active, these are the ones who are active and approved. These are the ones who are pending approvals and they could look at their contact information so that they can go look them up in their school database and, they do things like make sure that they don't have any disciplinary actions against them or make sure that their grade point average is above a certain amount or whatever and then they could go. Yep. All these people are good. Thank you very much.

[24:21] So, some of that did require because we built that as a react app, some of that did require us to have drupal API S, you know, like the JSON API S and things like that. So I think my next question is, it's not completely a fair question because I know it's not apples to apples between the drupal seven site to the drupal 10 site. And there's probably some functionality differences that we weren't able to get to, but just from a amount of custom code standpoint, moving from 7 to 10 more custom code, less custom code about the same. Any idea I would say, you know, you'd probably measure it and it would be less. And a lot of it is actually in other parts of the site that are not the parts that we're talking about right now, sort of like the parts that display things about what the programs are, you know, when, when does it start? When does it end? What courses are offered during that program? There's all this sort of display logic if you visit site and you never hit the register button, all the stuff that you see when you're not logged in, that's where a lot of the custom code existed. And a lot of the sort of like re architecting of data and things happened was on the display side.

[25:32] And there is, there is even more complexity because they, like I said, at the beginning, they have a sister organization and that sister organization actually wants to syndicate some of the information, not from the back end database, but from the drupal site. Because the marketing team for one is writing all this great content, uploading all these great photos and videos and information and the course descriptions. And then the sister company wants to syndicate that content, but they need to translate it into, you know, Japanese Korean or Chinese and they need to, you know, customize it for their sort of local audience. So they would use some of it but not all of it. And they also were not getting, you know, 100% of the information.

[26:18] It was a whole other thing. We're not going to spend a lot of time talking about it, but that's where a lot of the a lot of the custom code that we removed came from that. And I would say also in the things that would have to like save and validate the information because they were originally keeping two copies of all the information between backend database, the front end database. Now there's only the backend database and a light, a light wrapper around it that we need to do the drupal stuff like the dashboards, for example. So if someone was embarking on a similar project, drupal seven to drupal 10 heavy backend integration, what, what advice could you give them? What lessons, what, what are the, what are the, the difficult lessons that you learned that you wouldn't want to see anybody else go through?

[27:05] Absolutely. And I actually took some notes for myself so that I could remember what, what things that I would want to say because I knew I would forget at the moment, I will say probably the number one thing was, be prepared and, and start forming the relationship with the client and the the other back end developers. In this case, they were in house developers that were maintaining the back end. We were doing the, the drupal site and give the, you know, give them the expectation that we're going to come back to you and we're going to ask questions and we're going to sometimes challenge, you know, things that you are doing or ask you to do a little bit of extra work because.

[27:49] You know, we want to challenge the complexity, right? We don't want to just accept complexity and say, well, this is how it is, everything is just like this and end up doing all this extra work when maybe a short conversation could have really, you know, handled a lot of that problem. So the communication and also, you know, not deciding, well, we're just gonna do what we do and we're going to finish it and if it works then it works and if it doesn't work, it's not our fault. Right? You don't want to have that pointing fingers moment. Um, when you have multiple different teams that you're working with and these teams could be inside your organization or like in ours, it was a vendor and a client and an internal team, right? Or sometimes it's multiple external teams. You don't want to get into that game. Like start working on the relationship as, as soon as you can and, you know, be ready and be flexible.

[28:40] That's a big one, another one that we probably learned and I wish we had more time to do was sort of create. More internal documentation for what that external API was doing. In this case, the the internal team was working on the API at the same time that we were trying to use it because all these projects sort of got approved at the same time. And like, yes, we're going to do this, we're going to upgrade everything and the internal API is getting rewritten while we're trying to create our first code for it. A moving target. That sounds like fun. Yeah. And that also meant that sometimes the API would go down.

[29:27] Because they're doing an upgrade or it's a Java application, we have to redeploy and so it's down for a few minutes in, you know, in the middle of the working day, which is when the other people that are working on it want to be working on it too. So I wish that we would have also done better, better things about writing stubs or test cases, things that didn't depend on the API being there in order to do something like, hey, we submit the application. If it, if it was in dummy application mode, submit the application and get back response or submit the application and get back, you know, failure or submit the application and get back. This status was wrong in the real world. What we ended up doing was a lot more manual of like, OK, I'm gonna fill out this application. I'm gonna set one of the fields to something wrong. I'm gonna submit it and make sure I get the right thing back against the real API because we just didn't feel like we had time to build all those other things. But I, I kind of wish that it did. Was there time available for automated tests?

[30:28] To write automated tests. I did not do a lot of automated tests, but I did a lot more PHP standing than I ever did in the past on something like this, those that level of like linting it and, and parsing through it and things really mattered a lot when it would say like there's a variable here that is never getting read. And then I could go to somebody while I was doing their code review and say, hey, you know, please please run this tool so that you can find the same issue that Stan just found. What level of PHP stan were you running it at? I know all the other PHP stan nerds like me out there. That's a great question. I want to say that I'm usually somewhere in like five or six land. Yeah, that's the sweet spot I think for, for drupal development, it's really tough to get past six.

[31:16] I mean, you're going to see a lot of the interval array issues complaints that like you're not sending the right, you know, data to this function, but it's not actually very well documented what those issues are. It's mainly a documentation issue and a lot of times it actually ends up being something where the documentation is inherited from drupal core, which means.

[31:39] You know, it's, it's not going to be fixed for a long time if ever, because who, who can document, you know, all of the various levels of a note object. It's, it's too dynamic but any who, any other lessons learned. Yeah. I would also say work on the parts that you can and you know, focus on getting a working product. I mean, I think a lot of these things are related to that, but especially when you're working with an outside API if you have an idea that something could change or they might add new features if, if this is a long running project, right? It took us over a year from when we first sort of signed the contract to when we deployed something, and, and deployed another thing, I mean, some of this was happening in, in, you know, real time that as things were changing, we had to be, be ready for that.

[32:33] And, you know, understanding that it was coming again, like communicating with the other team and making sure we knew when their release windows were and when they were going to be adding new features or adding new functionality that we were relying on, meant that we could spend, we could prioritize what tasks we were working on, we could say, OK, well, we can't finish this new feature with the API until the next deployment of the new API thing happens. So let's spend some more time on the dashboard or let's spend some more time on the REACT application or let's spend some more time on the other parts of the site, you know, that are not related to this backend integration and we'll come back to this next sprint. So slotting in time for changes and waiting on outside forces I think was important. So is it all done? Is it all launched? Are you on to another project? It's, it is, it is way launched. And this, this all launched, I think more than a year ago at this point.

[33:29] So you can definitely go and use it and see it. And I actually don't work for the agency anymore that, that originally built this, but this is a project. I put a lot of my personal time and effort and sweat and blood into. So I feel like it would be cool to talk about some of the lessons learned. Excellent. Well, always great to talk to you. I look forward to seeing you in October, right. We're the October 11th, 12th and 13th Pacific Northwest drupal Summit in Seattle. All right, Mr Price. We'll see you on another episode in the future. I'm sure. Yes. Thank you. Thanks for listening to the drupal Easy Podcast. Don't forget to check out all of our long form drupal training courses at drupal easy.com and stay tuned for the next episode of the drupal Easy Podcast.

[34:20] Music.