Can you hire ChatGPT as a PM consultant?

1. Introduction
2. Write a good prompt for ChatGPT
3. Hey ChatGPT, would this fit my project?
4. Reviewing the answers
   1. Question 1
   2. Question 2
   3. Question 3
   4. Question 4
5. My thoughts
6. Can ChatGPT interpret Sprint Data?

Introduction

For some time now, one of the hottest topics has been artificial intelligence. In addition to the controversies, doubts, concerns, uncertainties that this technology has sparked, especially since the release of ChatGPT, many people are wondering how to use it to improve the productivity of their work or their organization.

Today, I want to try to answer one of my own questions by running an experiment: Can AI, in its current state, provide support to professionals involved in project management? Can a large language model (LLM),in this case OpenAI’s GP, offer useful insights and therefore be considered a sort of “consultant” for Project Managers or professionals with similar responsibilities?

To try to answer these questions, I will present ChatGPT with a sample project, including its context, task management methodology, and constraints to be followed.

The example submitted to the model in the following chapters will intentionally contain some mistakes or “strange” management practices, in order to test whether the model is able to recognize these issues and develop a solution while considering the project context.

Since I recently wrote a blog post about Sprint Data, here’s the link, and how it can facilitate the continuous improvement process within a development team, I also want to see whether ChatGPT is capable of interpreting that data and acting as an additional support tool in this process.

Disclaimer: There’s no guarantee that the answers provided by LLMs are always correct. Always question what you’re told, just as you always question the advice given by the most experienced members of your organization 🙂

Write a good prompt for ChatGPT

Before starting the experiment, it is important to make sure of the structure of the “prompt”(the question to be asked to the model) it’s written in a way so that the model can respond as accurately as possible.

OpenAI has published a brief guide about this at this link. In short, we need to make the model impersonate as a Project Manager and be as precise as possible in order to avoid any ambiguity in the text we submit.

Given the fact that i am not a technical expert on the matter of AI, i link here a video that really helped me to understand how this technology works and how to use it.

Hey ChatGPT, would this fit my project?

Let’s take action. Below is the question I will ask ChatGPT to verify the response to the first question of my experiment (Can AI, in its current state, provide support to professionals involved in project management? ):

“Imagine you are a Project Manager working for a software development company called SuperWare. Recently, a major pharmaceutical company named MegaMed, with production plants around the world, has tasked SuperWare with developing a software application that tracks the stages of their drug production. You have been appointed as the Project Manager for this new development project, and your team includes two business analysts, four software developers, and one tester.

A fellow Project Manager, who recently completed a successful project, suggests that you execute the project activities as follows:

• Divide the project into 4-week iterations;
•  The analysts should be responsible for defining requirements, writing User Stories, and estimating them in terms of Story Points;
• The developers should be responsible for implementing the User Stories described by the analysts;
• Use the tester for functional analysis and have them perform testing only at the end of each iteration;
• Schedule a meeting every three months with the stakeholders to verify that the product being developed meets the client’s needs.
  

Considering that the product must be delivered within 12 months and that each country where MegaMed has production facilities has its own regulations regarding drug manufacturing, answer the following questions:

1. Is the approach recommended by your colleague appropriate for this project?
2. Are there any issues with the approach suggested by your colleague? If so, how would you address them?
3. What precautions would you take considering the scope of the project and the 12-month delivery timeline?
4. If you had to define a methodology from scratch for carrying out the activities of this project, how would you do it? What would you pay particular attention to?”

Reviewing the answers

ChatGPT responded point by point to the questions we asked, so for convenience, let’s analyze them one by one.

Question 1

Is the approach recommended by your colleague appropriate for this project?

Here, ChatGPT answers correctly although somewhat optimistically. The methodology recommended by our colleague is not partially correct, it simply doesn’t make any sense. That aside, the model correctly identified the key points of the project:

• The product will be used in different countries and therefore must comply with various regulations;
• The product must be delivered within 12 months;
• The product is intended for a pharmaceutical company.

The model states that a more rigorous project management framework is necessary and elaborates on this point in its responses to the following questions.

Question 2

Are there any issues with the approach suggested by your colleague? If so, how would you address them?

The example approach provided to ChatGPT was intentionally written to be inefficient and incorrect in order to verify how many errors the model could identify, along with the corresponding solutions.

1. ChatGPT begins by stating that meetings with stakeholders should be held more frequently in order to gather as much feedback as possible. In this case, the model suggests holding a review of deliverables at the end of each Sprint (every four weeks). It also recommends including at least a subset of stakeholders, especially those with expertise in legislation relevant to the project. I would add that, in this case, it is particularly useful to define a RACI matrix to clearly identify which individuals should participate in the Sprint Reviews or be consulted during the implementation of specific functional requirements.
2. There isn’t much to add regarding testing activities: as correctly pointed out by ChatGPT, testing should be performed continuously in order to detect issues as early as possible.
3. Another correct response: analysts should not estimate tasks on their own, as the estimation process should be carried out by the entire team to properly assess the feasibility of a user story. In this case, we also have Quality Assurance professionals available, whose input is essential during story estimation to account for the time needed to write and execute test cases. If these aspects are overlooked, the resulting estimates may be incomplete and lead to inaccurate development forecasts.
4. The methodology proposed by our colleague does not include any guidance on how to manage multiple legislation within a single product. The model suggests having one epic per legislation and building a modular system that supports the product’s use under various legal frameworks. This point doesn’t entirely convince me. In this case, the decision to create one epic per functionality is closely tied to the technical and functional approach chosen for the project’s implementation, and should therefore be made in alignment with the entire development team.

Question 3

What precautions would you take considering the scope of the project and the 12-month delivery timeline?

ChatGPT provides a number of useful suggestions, but in this case, I believe it misses a couple of key points:

1. There is no mention of protecting the project scope. Given the limited resources available, both in terms of personnel and time, it is especially important to avoid “scope creep.” Particular caution should be exercised to ensure that the project stays focused on the originally defined goals without adding new unnecessary or lower priority requirements that could compromise delivery.
2. There is also no mention of prioritizing development activities. In this context, the “highest risk” and “highest value” task is implementing workflows for different countries based on their specific legislation. In my opinion, the team should receive feedback primarily on these activities so that any necessary corrections can be made as early as possible in the schedule, even if that means dropping other lower value features.

Question 4

If you had to define a methodology from scratch for carrying out the activities of this project, how would you do it? What would you pay particular attention to?

Here, ChatGPT essentially outlines a project management methodology that could be adapted to the context we provided. However, the question may have been too vague, and as a result, the response was also somewhat generic: the model simply provides a list of steps to follow, but doesn’t explain the rationale behind them. Beyond a few points I disagree with, for example, the length of Sprints in a project where the risk is high and time is limited, applying any of these options without understanding the reasoning behind them wouldn’t likely bring significant improvements compared to doing things randomly. At this point, it would be interesting to ask ChatGPT why it gave that specific answer.

Unfortunately, when I submitted the same question again, I received a different response.

This reveals both a particular strength and a somewhat subtler, context-dependent weakness:

• The fact that the answers are never exactly the same can help a team brainstorm around different topics;
• However, I think there could be the “human” tendency of not examining the responses and combine multiple of them in order to feel confident that a “comprehensive” methodology has been implemented. Clearly, the model itself isn’t to blame, it simply answers based on the input, but this is something to be mindful of when considering the integration of such technologies into an organizational context.

My thoughts

ChatGPT gives us a brief summary of the previous responses and asks whether we’d like a project plan for the next three months. Not wanting to turn this experiment into an endless exercise, I didn’t follow up with that question, but it will definitely be interesting in the future to explore the model’s planning capabilities as well.

I have to say I was pleasantly surprised by some of the answers provided by the model. What I question isn’t models’ ability to interpret the question, but rather the ability of the person asking, myself included, to provide enough context for the answer to be as detailed and targeted as possible. For example, since I didn’t mention how many countries are involved in the project, the response didn’t raise any concerns about whether the project could be realistically completed within the available time and resources.

A question with some oversights, like the one I asked, could lead the model to give a response that isn’t complete or fully appropriate to the situation. If such an answer were applied without critical thinking, it could lead to serious difficulties in managing project activities.

In conclusion, I’m satisfied with the model because it offers very interesting points of reflection. This first experiment is, in my opinion, a success—despite the concerns I mentioned earlier, because ChatGPT was able to identify the key issues in the question I posed and point me, albeit briefly and with solutions still needing validation, in the direction of possible resolutions.

I’ll try to use ChatGPT as my personal “PM Consultant” in order to evaluate this experiment better by using real cases. At least for now, i’ll try to ask to the model questions where I’m less likely to omit key details that could easily lead to inaccurate answers.

Can ChatGPT interpret Sprint Data?

As the final part of this experiment, I wanted to try providing the sample data I used in the post to see if ChatGPT can be a useful support for the development team.
The question will contain no project context, I’ll simply provide the sprint data and ask the AI to take on the role of a team developer and identify areas for improvement.

ChatGPT says:

ChatGPT provides an analysis of patterns and bottlenecks that can be identified through the sprint data we provided. For each Sprint, it highlights key “pain points” and offers targeted observations.

It then identifies the main issues found in the sample data, most notably, the model correctly points out that QA is struggling to keep up with the developers’ velocity. It also suggests a series of actions that could help mitigate the issues identified.

I get the impression that, considering how well these tools can process large volumes of data, LLMs are already ready to be used and implemented in a wide range of project contexts for this type of activity.

I see the model’s response as a kind of “guiding outline” that can support the team in identifying problems that might otherwise go unnoticed, especially when there’s a large amount of information to process (eg. Sprint data of 100 Sprints). In addition, the tool could also be used at the end of a project, by feeding it all the recorded Sprint data and using it, together with the project team who carried out the work, to generate a report, document lessons learned and identify potential solutions that could have led to a smoother process but weren’t spotted during execution.