… testing is unavoidably ‘all over the place’ (seven notebooks around me at the moment), with just one good place to gather and improve information. … lose track of some things …
A few days ago I found, maybe in Gitter, some nicely worded hints about approaches to reporting issues. Such hints are not unusual but as there’s extensive use of Gitter for TrueOS, it’s worth drawing attention to what was suggested from a Gitter perspective.
Unfortunately now I can’t find those hints. Yeah, that’s what happens when things are all over the place so please be patient.
As someone who has been playing at software development for a long time, the worst thing that could happen was have QA file a bug. Then you go talk to the filer, ask simple questions “what were you doing?” and get “I don’t know. It was just sitting there and crashed”.
The best thing was to have QA file a bug “I started from a cold reboot, verified the system was up and running correctly via the following commands, then I started the application, clicked File, Open and it crashed. And here’s the core file”.
Sometimes with a general purpose OS, it’s difficult to keep track of who what when, so at least making an informal note of bad behaviour is helpful in case it happens again.
Not being privy to the gitter conversations, I’m simply offering my experience from the dev side, because the dev job becomes easier the more information is available. But I recognize that QA is not a fun job either.
Finding most of the dates wrong was a nasty surprise, that’ll not happen again. For systems testing I normally begin the names of files with a relevant date and time, so for TrueOS bug reporting purposes the wrongness is not a major setback.
Things stay on my desktop for not too long, I move them to a relevant folder but not necessarily the working folder for a particular problem (or particular piece of software).
For the first screenshot I selected a bunch of network related stuff, including a pc-netmanager folder. Some of what’s selected belongs in that folder, no rush to put it there because of most of what’s wrong with Network Manager was observed months ago; bug reporting was deferred.
It’s all to often when we get bug reports of ‘$thing doesn’t work’. Or for people to find that something doesnt work properly and before diagnosing it, they run to another system and then say “well I couldn’t reproduce it on this machine.” That annoyed me so much in Linux circles.
When someone finds a bug, the first thing they need to do is document every step they took which caused it to happen on that machine. Adding another machine into the mix just complicates things needlessly. The biggest help they can provide is to figure out the steps needed to reproduce the issue. While debug logs are helpful, having a clear concise path to reproduce the issue is the greatest help. Logs can give an idea of what was going on in your system, but having a step by step procedure to recreate the issue helps far more.
Since individual systems vary so much, knowing the step by step procedure to reproduce an issue is far more helpful because it allows the developers to start eliminating possible reasons for the issue.
For anyone reading this in the future, the most important pieces of information to resolve an issue us are:
What happened and how that was not what you were expecting.
Steps to reproduce the issue
If you have changed your system in any way from its initial install, what have you done?
What is the hardware of the system where the issue occurred.
Where/How people save this information before submitting it is up to them. I know some prefer text files, others prefer using a google doc of some sort, others just email the information to themselves. None of that really matters to the devs, as long as when you eventually do present the information it’s in a clear and concise manner. Since we cannot read your mind, we are dependent on you explaining everything clearly and not expecting us to think the same thing that you do when we see whatever material you are presenting. Case in point, if you’re sharing a screencap because its easier to explain a problem, that’s fine;. But be sure to explain exactly what in the picture you want us to notice, because we may not be able to tell what the problem is. If you need a free place to host screencaps, http://imgur.com works well. Lastly, dont think that a screencap is a replacement for the 4 pieces of information listed above.
Imagine trying to do that with … an operating system that typically could not save data to a USB flash drive; with an OS that’s based upon an OS with imperfect or (ahem) no decent support for some very commonly used network hardware; an OS that will not let me perform a right-click with some hardware; an OS that – during the transition to OpenRC – would not let me use a keyboard or mouse; … I love TrueOS Desktop but bleeding edge, real world, in-the-wild testing of the OS in situations such as those can be quietly exasperating.
I can nearly always laugh off such glaring deficiencies in the software, but to put it bluntly:
in situations where information can be neither (a) easily saved to disk nor (b) easily transmitted over a network connection, I expect a reasonably relaxed response from recipients of that information.
If information that’s gathered in such laughable environments is sometimes delivered in a piecemeal manner, accept it with thanks and with grace. Put yourselves in the shoes of testers who have quietly jumped through hoops to get things together.
If I may jump in here, the information from steps 1-4 can be easily transmitted from another system (if available). Heck, I still keep spiral bound notebooks and pens so I can scribble notes by hand. If needed I could transcribe that from a Windows PC into an email or a post to this forum. Kind of like “My car won’t start” vs “My car won’t start, I have a full tank of gas, it makes a click-click-click sound when I turn the key and I stuck a DVM on the battery and it reads 12.3VDC” to your mechanic.
I also recognize the frustration of a system that was working until I did and update and now it doesn’t work; the systematic gathering of information as to what not working means helps everyone out in the long run.
For me the greatest problem with failures at a basic level (e.g. inability to use a USB flash drive) is that each and every such failure disrupts the train of thought.
I sometimes want to not jump through unrelated hoops to record a problem with a hoop. I lost count of the number of bugs, some of which were serious, that I chose to not report because trains of thought were disrupted.
I’m not trying to be a jerk here, but how is asking you to write down 4 simple pieces of information on a piece of paper (as mer mentioned) and submitting it on a good working system causing you to jump through hoops?
It seems to me, for whatever that’s worth, that you’re actually causing yourself more headache by complicating the issue because you’re trying to do more than is needed. By trying to capture logs, and picture screen captures, and video screen captures… you’re increasing the number of ways that you’re going to have trouble. Reporting those four details is FAR more helpful than a video screen capture of whatever. If we need a photo or a video, we’ll ask for it.
If a problem occurs, stop and make notes about that single problem. Dont try to rely on something else which may or may not work as well. What I’m asking people to do is to vasly simplify their technique. Not only will this result in higher quality material for the devs to work with, but also means that each individual issue can be treated on its own and not as one piece of a ten issue nightmare scenario. On top of that, since you are focusing on one thing at a time, hopefully you wont get frustrated as issues cascade.
Here are a couple examples of good reporter, and developer workflow:
In each case the user gave only the right amount of information to reproduce. Although I am guilty of adding a few links in #181 while I was looking to find the cause of the issue he followed up very quickly with feedback on those links. He was reading it as I was. In this case the links were directly related to the problem we were seeing, and contained no other content.
So now we have 92 some bugs which we all know are mostly from one person. Thank you @grahamperrin for the work you have put into testing. I know you most have spent a ton of time on what you have done.
Here I think we had a good workflow going:
Until "The two deletions had a positive effect for a different issue; discussed around " in #192.
Now I am just lost on that one. My brain stops working as I begin to scroll up, or down to find what I am trying to follow just to determine how this relates to what I was just reading. On that point I can see what @q5sys is talking about. It’s not the same as a good old fashioned dialogue to solve a problem. A simple summary would have went a long way in trying to pick up where we left off.
There’s now a postscript, which may help to clarify things.
Going off-topic in a tracked issue sometimes displeases readers. I try to avoid it.
At a more basic level, there’s sometimes discouragement from using trackers for dialogue or discussion. Use of Gitter for discussion can help to respect the people who prefer tracking to be as terse as possible.
It’s a useful discussion, but this is not the place to continue. Please join me in the lounge.
– and here in the lounge, the first screenshot (above) shows that I’m already in the habit of using folders for issues. I have been doing so for years.
It’s difficult, sometimes impossible to please everyone.
Reflecting on experiences with Apple
Apple Developer Connection
I became a member of ADC Online probably in 1999. Not as a developer, I joined primarily to report bugs. Captured in 2008: Apple’s Bug Reporting Best Practices – the essence there is not unlike the essence when I joined.
https://appleseed.apple.com/ – not to be confused with public beta testing, which began years later in the same area. When I was invited, black label DVDs were the norm for distributions and over a period of around five years I gave feedback on a number of projects – including highly-focused and fast-moving projects for early versions of Feedback Assistant, the client application that later helped to streamline things in much larger private projects such as those for Mac OS X and Mac OS X Server.
I’m not here to break confidentiality, just a handful of points:
for bug reporting practices, presentation of guidance to testers was quite different from guidance to developers
as far as I recall, Apple never asked me to refrain from any particular style of reporting.
Still, I knew that I could do better and so I once asked a member of the Seed Team what I might do better. The answer was memorable (and I had no argument): the wish to receive, from me, more timely responses to Apple’s requests for feedback.
Graham, this is exactly the linking issue which I’ve been trying to explain. I realize in your head what you wrote makes sense, but for the rest of us who aren’t in your head it doesn’t.
When reading through the initial bug report we have no way to know or understand how the information you are linking is relevant. On top of that, when you what you are linking can’t be fully understood without following another link, we are sure not to understand what you are trying to explain.
There have been several issues you’ve brought up in the past, but I simply cant make sense of what you are trying to explain. Again, I know this all makes sense in your head, but I’m not in there with you mate. I dont know the jumps you are making between points because you’re not explaining them. You usually make a comment like ‘further consideration’ or similar. You know how it’s similar because you have already read it and come to an opinion on the material. Since we dont know what you have concluded from the attached information we can’t follow your thought process.
Example: In your post here. The thread seems to be about a wifi dongle, yet you decided to attach a link to the poweroff button issue? I cannot understand how the ACPI power button issue you linked from github is relevant to a wifi dongle.
– If it’s unrelated to the wifi issue, that’s the type of information that should not be linked because it will only act as a distraction.
– If it is relevant… you haven’t explained why you think it’s relevant and worthy of attention while we try to resolve the dlink wifi problem you are having.
Situations like this leaves us with two problems. 1)Trying to figure out the software issue which you are reporting. 2) Trying to figure out what you are trying to explain with other things you have decided to link.
I appreciate the amount of effort you put into reporting issues and I’m certain the rest of the dev staff feel the same way. I know you want to help, and though it may not seem like it, I’m trying to help you be of more help. I have been trying to get across that if you change a few ways of reporting things, your help will be so much more meaningful and have a greater positive impact on the project and the community. I feel like you get frustrated by having to continually respond when you are not understood… so lets try to avoid that in the future. The way you wrote your post script on that linked github issue is excellent, and I’m sure had you written that the first time Joe would probably have been able to follow along a bit easier.
It’s true that a lot of the times issues do overlap, in those cases… go ahead and make a second issue for the related item. It will get more attention if it’s treated as its own issue, since it allows us to focus on a single issue at a time. If you want to cross link them, a simple blurb at the bottom of “this may be related to $X” would be fine, but make sure that it’s not required information to be able to deal with the issue that’s in the title.
Regarding the apple thing you linked.
Most of what they ask for is similar to what I’ve listed above in those four items. Due to the smaller nature of our project, I would say hold off on crash logs and screenshots unless we ask for them. I would say that in most situations we prefer Functional descriptions over Non-functional descriptions; to use their terminology. But I’d agree with everything in
[quote]2.2 : Description:
Include a Summary, Steps to Reproduce, Expected Results, Actual Results, Workaround, and Regression/Isolation.[/quote]
For context… compare this excellent report you made with this one. I’m not only referring to the initial report, but once you had time to investigate and then report back.
After thinking about it a bit, I think it might be a good idea for us to write up an “Issue Reporting Best Practices” guide. I’ll start working on one, and discuss it with everyone in TN when I’m back down there in a few weeks.
Readily available guidance is a good thing but bear in mind, it will not suit all head spaces, all situations. Please be flexible about what’s reported in trackers and elsewhere; bug reports, enhancement requests, questions – whatever the nature of a post, it’s nearly always true that if something is not clear, a handful of questions will lead to a shared understanding. Often true that a single question will suffice.
Never frustrated by simple misunderstanding and I certainly don’t think of misunderstandings as continual.
Essential to how I work (and this could be true for any writer):
If a report with fifteen steps to reproduce a bug can be excellent, then – for a simpler bug – a report with four steps (and a single screenshot showing the bug) could have been lovelier. A summary comparison:
– and so on. For Android I can find nothing as superb as Mactracker, the next best thing is probably EveryMac.
Wherever I mention things such as MacBookPro8,2 or MacBookPro11,2 that is the concise way of identifying a model without repetitively adding unnecessary detail.
Please note, Apple’s document for MacBook Pro models includes a few errors. For example, MacBookPro11,2 (a true identifier) appears in the screenshot but not in any of the tables that follow. In this case the misidentifed model can be found by seeking 11,2 within the document.
Please do not conflate the word count of a ticket to whether its written well. A ticket that has tons of words but that are all directly related to the issue is better than a short ticket that doesn’t provide the needed information. Conversely, a short ticket that’s directly to the point and doesn’t include a ton of unneeded information is better than a long one that tries to relate to every thing on the planet.
What’s important is staying directly on point and not wandering into other things. A footnote of, “could this be related” is of course fine, but the bulk of the bug ticket needs to be directly related to producing/resolving the bug. It’s best for everyone to not wander to far into related things unless asked to do so.
Can we come up with a list of commands to get the information (or at least most of it) you’re looking for?
OpenRC scripts in sysinit and shutdown runlevels could push info to a log file to cover most of point 1. Capturing the update would likely require tying into sysadm/pc-updatemanager, but since the updates are already shown on the console when you restart, that should be doable. /var/log/pc-updatemanager.log has all the update info in it.
beadm list gives clear data on items 2 and 3 (name and date of creation)
uname -a gives more than uname -v
uname “version level” ties more to “how many times has the kernel been built” and what is the source tree and the config file used. So, unless one is building their own kernel packages, this should tie closely to a boot environment.
This may be fun. pciconf can likely get a bunch of info (pciconf -lv, look for items of class display).
“which ones in use”, that would mostly tie into the X log I think, also would lead into the drivers in use, maybe not the user selected ones, but they should match.
Output of kldstat is nice to see what modules are loaded, let’s you figure out if the right thing is loaded for your device.
I’m not sure of how to get available display rendering technologies.
uname -v tells you the branch and commit level of the source tree the kernel was built on, not specifically a display rendering technology.
Wayland; isn’t that more related to X (or not X)?
The Periodic scripts can be configured to stick log files into /var/log instead of mailing the output to root (I find it more useful to have logs). They may have some information you want already.
If you/we/all of us can come up with existing commands to get you most/all of what you want, I think they could be munged in an OpenRC script that can run in sysinit and shutdown, heck or even called from periodic or sysadm-client.