Coloring plastics at the press is one of the most functional, value-added features a molder can impart to a molded product. Self-coloring not only improves aesthetic properties, it can also Improve UV stability and make processing easier. Also, adding color at the machine is almost always less expensive than purchasing precolored material from a supplier.

There's just one catch. Once the color system has been incorporated into the polymer matrix, it becomes an integral part of the material and may alter its engineering properties as well as its processability. As a result, it's important to be aware of some of the common problems involved with melt-coloring plastics and how to avoid them.

Generally, aesthetic flaws can be attributed to three different causes: equipment (machine, mold, ancillary equipment), molding (process), and design (part and tool)/formulation (base resin and colorant). In some instances, the flaw may be the result of problems in more than one category, but generally, there is always a predominant issue. Similarly, some problems may have more than one symptom, so several different "problems" may be showing up in parts that have the same root cause, which calls for the same corrective action.

When troubleshooting aesthetic problems in molded parts, it's important to begin with an open mind. Since successful molding is an interdependent combination of machinery, labor, material, and design, troubleshooting is often more complicated than people initially expect.

Identifying the Problem
The first step in determining if the flaw is caused by a molding- or colorant-related problem is to inspect parts visually to determine what the problem looks like. Some flaws are very distinctive and quickly lead you to their root cause. Others are less distinct and may be caused by a number of factors that fall into several categories. Consequently, some basic troubleshooting and root-cause/corrective-action work is required. The table with this article provides a list of common aesthetic flaws that could involve colorant and their most probable causes.

The next step is to find out when and where the flaws are being reported. Flaws that show up immediately upon molding require one approach to troubleshooting-for instance, processing and equipment issues are easier to check at this point. In this case, the troubleshooting process might start by asking the following questions:

• Have the process settings changed?
• How frequently does the problem occur?
• Is the proper equipment in place and functioning?
• Was the colorant properly metered in and mixed with the melt?
• Is the colorant compatible with the base resin?

However, field failures that occur after months or even years will suggest a different starting point-a focus that begins with either part design or formulation:

• Is UV exposure higher than initially predicted?
• Is the part being used as originally designed?
• Has the part been inadvertently exposed to some chemical agent?
• Is the aesthetic change a symptom of a larger problem (for example, loss of mechanical properties)?

Regardless of the simplicity or complexity of a molding project, good documentation practices should be in effect and followed diligently by all team members. Processing parameters should be noted throughout. Not only will this practice help molders stabilize their process and create good parts faster, it will also provide a baseline for reference should the process drift and problems arise.

Common Sense- Often Uncommon
The best way to begin troubleshooting an aesthetic problem in which colorant and/or special-effects packages are involved is to rule out the colorant package as the source of the problem. This is done by running uncolored (natural) material through on the same process settings to see if the problem recurs. If it does reappear, then it's likely that the problem is not caused by the colorant package, although the base resin could still be part of the problem. If the problem disappears, then the colorant package is the proper place to begin your investigation. It's important to remember that aesthetic issues may exist that are difficult to see without color. In other words, a colored background will frequently magnify aesthetic problems.

While it is outside the scope of this article to cover the entire process, one thing that should be mentioned is that proper troubleshooting requires a sensible, methodical approach. Unless a Taguchi design of experiments study is being conducted, only one parameter at a time should be changed, and its effect should be carefully evaluated and documented. Then the parameter should be returned to its baseline setting before another parameter is changed.

While the following guidelines involve colorant-related problems, we will also touch briefly on processing, equipment, and design issues, indicating areas of concern that can affect colorant and/or aesthetics. If the troubleshooting process indicates that one of these categories is a likely cause, then there are a number of good resources, both human and written, that can and should be consulted for further help.

Troubleshooting Guidelines

As with any manufacturing process, molding a quality product consistently requires proper equipment, design, material, and processing procedures. The following list of considerations are provided as general guides when beginning to troubleshoot aesthetic problems.

1. First, check the following four equipment areas.

Screws/nonreturn valves. In most cases, a general-purpose screw will work well with colorants, depending on the letdown ratio, chromaticity, and base resin used. Although it's usually not necessary, a growing number of injection molders are using mixing screws to aid in color distribution. Processes with high throughput, short cycle times, and large shot sizes or those that use difficult-to-color resins will often benefit from this type of screw.

As with screws, selection of non-return valves is often resin-specific. Ball check valves are more shear intensive and can aid in color distribution, although this method can prove problematic when processing parameters such as screw speed, backpressure, and barrel temperatures are pushed close to the base resin's limits, leading to both colorant and resin degradation.

Screw and barrel wear. A worn screw or barrel-regardless of the design-will inhibit mixing, causing a loss in plasticating quality and therefore color distribution.

Barrel capacity. As a general rule, using more than 60 percent of the barrel capacity with a GP screw is considered to be pushing the processing envelope for good color incorporation. Of course, this is cycle-time dependent. The barrel's L/D ratio is also important. For example, shorter barrels (18:1) are less conducive to a good mix, while longer barrels (24:1) provide a longer mix environment, but also longer residence time. It's important to keep an eye on material residence time, because excessive residence can degrade either the base resin or the colorant, or both, while insufficient time in the barrel may lead to distribution issues.

Colorant handling and dispensing. One of the keys to successful color processing is an accurate blending/distribution of colorant and base resin before both enter the molding machine. When blending equipment is used, production personnel must be aware of proper operation, calibration, and maintenance. If the equipment is not properly calibrated to the process, any money saved by coloring the raw materials will be lost in wasted product.

For solid (salt and pepper mix) colorants, there are many different options, such as drum tumblers or auger mixers. For blending concentrates and powders at the machine, either volumetric or gravimetric feeders are used. Volumetric feeders are both less expensive and potentially less accurate than gravimetric types because they feed colorant based on speed vs. time. Gravimetric feeders weigh and blend each component in the mix. For liquid colorant, special volumetric pumps are typically used to handle the color accurately and neatly. (See the sidebar on p. 87 for definitions of common colorant-system formats and their functions.)

2. Tooling design can also impact aesthetics.

Gates and runner systems. Proper gating and runner types should be considered at the design table. It's important to remember that additional heat generated by a hot manifold or pin gate during injection fill can affect the integrity of the color. If it's known ahead of time that such tooling is being used, a colorant provider can formulate the color package differently.

Knitlines. Anticipating aesthetic issues during the early design stage is very important, especially if knitlines are a factor. When moldfilling analysis or experience indicates knitlines are likely, color formulation changes can be made with certain resin families that will Improve the appearance of knitlines. Of course, this issue is more easily resolved during the early stages of part and tooling design.

Mold texture. Also an important consideration during the design stages is mold texture, since it can affect how the eye perceives color. The presence of gloss and/or texture in the tool can make what would otherwise be an exact color match appear very different. Gloss should be considered in relation to both the mold and resin being processed, since some materials naturally have higher gloss.

3. Take a good look at part design.

Color/special effects. It's important to understand that a joint blessing and challenge of molding color and/or special effects is that the color package may accentuate design or molding problems that are already present in a part, such as knitlines, slight sinks, or stress-whitening marks near the base of a snapfit. Taking color back out and running the part in natural resin may appear to make the problem disappear, but in reality the problem, if it is a molding or design issue, is probably still there, and just not as visible.

The problem will be accentuated in chromatic colors and high-gloss designs. So if it's something that can otherwise be lived with, slight design modifications may solve the problem. On the other hand, certain problems, like stress-whitening marks, can often be hidden by the correct use of color or possible reformulation.

When integrally colored, molded parts of dissimilar materials are mated with other molded or painted parts, they must often match in color, gloss, and texture. Here, it's important to work closely with the resin and color suppliers to ensure that the parts will match as closely as possible when they are new and that they will fade at the same rate throughout the lifetime of the assembly.

Bosses/ribs/knitlines/parting lines. Wherever there is a flow interruption in the part design-from bosses, ribs, or even converging flow fronts on multiple gated parts-there is a strong possibility of sink marks, knitlines, and compromised mechanical integrity because knitlines are always weaker than the surrounding resin. Since the orientation of the material in these areas shifts, the refractive index across this region of the part will also shift, posing aesthetic challenges for many special-effects packages. Therefore, these interruptions should be designed into nonvisible faces and noncritical areas of the part.

4. Take these processing tips to heart.

Screw speed. For color processing, a slower screw speed is usually better to avoid shearing and burning the colorant and to ensure proper color distribution, cycle time permitting.

Backpressure. Higher backpressure promotes more thorough mixing, especially with a GP screw, again, if cycle time permits. A maximum value of 300 psi (2 MPa) is generally recommended by machinery OEMs.

Resin temperature. Higher melt temperatures also promote better mixing, as long as the processor is careful not to begin melting too early in the feed section of the screw, although this too can be resin dependent.

5. Be aware of these colorant and formulation issues.

Resin and colorant compatibility. The melt index relationship between the base resin and color are very important and should be considered early in the design. Additionally, certain pigments react with particular resins to a greater or lesser degree.

For instance, some organic pigments tend to dissolve and disappear (or change color) in styrenic polymers. Similarly, many dyes tend to change color or disappear in nylon polymers, which are weak reducing agents that effectively destroy the chromophore component of the dye that provides color. Additionally, except in special cases, dyes are not recommended for use in olefins because of very limited solubility and a potential for color migration (crocking). Further, dyes can act as a plasticizer in some materials, reducing thermal properties or even reacting with the resin itself to produce various effects.

Phthalocyanine (phthalo) pigments can be problematic, since they can act as nucleators, causing often uncontrollable shrinkage and warpage problems in crystalline resins. Fluorescent dyes also can pose problems in certain environments. They lack thermal stability for use in many engineering resins, they chemically react with nylon, and they lack UV stability for use in most outdoor applications.

A final category of problems is often found with polymer blends and alloys. There can be phase boundaries between the various components of these materials that can cause light scattering and a resulting milkiness. Hence, obtaining chromatic colors can become a costly undertaking due to the large amount of colorant that needs to be introduced to counteract the milkiness-often at the expense of economic concerns and critical physical or mechanical properties.

Carrier specification. The specific carrier for the color and its flow properties vs. the base resin play an important role in color distribution, and must be considered during the initial design. The carrier should melt in the late-feed or early-transition sections of the screw to ensure proper distribution of the pigment or dye in the mix. If the carrier has either too high or too low a melting point, the colorant will not be properly distributed in the melt, and finally in the molded part.

Other additives. Special additives for the colorant package can help with some color-related problems. However, these ingredients are normally not incorporated until processing efforts have failed, primarily because they add cost that may make the end product less cost-effective.

Grade can make a difference. It's important to remember that not all resin families or even grades within the same family are interchangeable with a given color package, let alone with different pigment or dye packages in the same chemical family. Further, since most colorant suppliers custom formulate their packages to a specific resin grade by a particular manufacturer, if the resin provider is switched after the color package has been formulated, it should be retested.

A good example of this is the fairly wide fluctuation that can be seen in the natural color of polypropylene, which can range from a yellow-white to a gray-white shade, depending on the quality and quantity of filler used.

An Ounce of Prevention
Like so many other problems in life, colorant issues are far easier to anticipate and prevent than to cure once they've occurred in molded parts. If color is important to the final performance of an application, then it should be designed in from the start via simultaneous engineering. To ensure the colorant package is properly formulated for the application, provide the color formulator with all available information on both the application and its material targets as soon as they are set.

If any of these specs change during the latter stages of the design process, inform both the resin and colorant suppliers. Changes may require a new formulation. Don't hesitate to call on the colorant provider to analyze, retest, and reformulate a color formulation to make sure it will be suitable in the end-use environment.

It's also important to inform the colorant formulator of your equipment setup, in case you need to make changes in order to optimize the resin/colorant/throughput requirements of a particular application. Finally, watch your process parameters and once the process has stabilized, note any sudden changes that may signal the beginnings of a problem.

Special care is needed when self-coloring engineering resins because a significant portion of their perceived value-and therefore cost-lies in their superior physical, mechanical, and aesthetic properties, which can be sensitive to even small compositional changes. The more important color is in a given application, the sooner the colorant provider should be involved in the design process.

Aside from that, you could uninstall the app, reboot and then install it again.

istockphoto.com

Q: I just opened the dishwasher door to find the bottom of the unit filled with dingy water. My first thought was that the cycle didn’t finish so I ran the dishwasher again, but that didn’t help. Why is my dishwasher not draining? Is this a do-it-yourself fix, or do I have to call a plumber?

A: You did the right thing running your unit a second time. If it gets inadvertently shut off during a cycle, there’ll be standing water in the dishwasher bottom when you open it. When that smart move doesn’t solve the issue, the problem lies elsewhere. Dishwasher service calls are common in the plumbing industry, but fortunately, the fix to a dishwasher not draining is often something simple you can do yourself. Before you call a pro, troubleshoot your dishwasher by following these steps.

Determine the type of dishwasher drain connection you have.

The first step in troubleshooting a dishwasher that’s not draining is pinpointing the type of drain connection beneath the sink. Open the cabinet and look at the configuration of the corrugated dishwasher drain hose and how it connects to the sink drain. The following drain connections are the most common.

• High loop to disposal: In this configuration, the dishwasher drain hose will enter the cabinet from the dishwasher—loop to the top of the cabinet, where it’s held in place with tape, a string, or a chain—and then connect to the garbage disposal.
• High loop to drain: If no garbage disposal is connected, the dishwasher drain hose should still have a high loop, but it will then connect to the drain pipe. The connection must be before (above) the P-trap on the drain line. Sewer gases can enter the dishwasher and leave a stinky smell if it isn’t.
• Air gap to disposal: In this configuration, the hose runs to the top of the cabinet just as in the high loop, but instead of having a tape or chain to hold it in place, the hose connects to a fitting at the back of the sink. If the dishwasher drain connection has an air gap, a small cylindrical knob will be installed on the sink’s top to allow air to enter and prevent dirty sink water from draining back to the dishwasher. In essence, an air gap is like a dishwasher overflow that prevents air locks in the hose.
• Air gap to drain: This is the same configuration as air gap to disposal, just minus a garbage disposal. The hose will run to the air gap and then connect to the drainpipe before the P-trap.

Run your garbage disposal.

The drain hose from your dishwasher empties into the garbage disposal drain. If the disposal unit contains unground food, or food sludge has settled in the drainpipe below the disposal, it can prevent even the best dishwasher from draining properly. Sometimes, running the disposal is all it takes to get the dishwasher draining again.

In fact, it’s a good idea to get in the habit of leaving the water on and letting your garbage disposal run an additional 15 seconds after the food is gone. This clears all remaining food that might otherwise remain in the P-trap drain beneath the disposal.

RELATED: 9 Tips for Keeping Your Garbage Disposal in Top Shape

Remove standing water in the bottom of the dishwasher.

If running the garbage disposal doesn’t help (or you don’t have one), it’s time to drain the water in your dishwasher so that you can take a closer look at what could be causing the problem. Place absorbent towels around the base of the dishwasher and then remove the bottom dish basket by simply sliding it out. With the basket out of the way, use a plastic cup to scoop the dirty, foul-smelling water into a bucket for disposal. When the water level is too low to scoop, use towels to sop up the last bit in the bottom of the machine.

Clean your dishwasher air gap.

In most cases, a drain connection with a dishwasher air gap is the least likely to have problems, but whether yours has one depends on local building codes. Some communities require air gaps, but others require only a high loop under the sink cabinet in the hose. If your dishwasher has an air gap, it will typically be located next to your kitchen faucet (in the photo at right, the air gap is to the left of the spray nozzle).

Occasionally, an air gap can become clogged with debris, which can prevent the dishwasher from draining correctly and leave standing water in the dishwasher. To investigate, twist the air gap counterclockwise to remove it, and check the inside for gunk. Clean the air gap with water and a stiff brush, replace it, and run the dishwasher cycle again.

Clear up detergent mishaps.

Dishwashers are designed for use with automatic dishwasher detergents that clean without producing suds. In the course of a hectic day, it’s easy to accidentally squirt regular dishwashing liquid into the unit, which can easily create enough suds to prevent proper draining. The same problem can occur if you run out of automatic dishwasher detergent and substitute laundry detergent in a pinch. Guilty as charged? Simply bail out the tub as described above and run the cycle again, this time using the correct dishwasher detergent.

RELATED: Solved! 4 Reasons Why Your Dishwasher Isn’t Drying

De-gunk the drain basket.

The drain basket is often the source of a smelly clogged dishwasher. It’s found at the base of your dishwasher, usually below the bottom rack. Its cover often resembles an upside-down basket, which either snaps off or is held in place by one or two screws. (If the bottom of your machine doesn’t look like this consult your owner’s manual, which you can often get from the manufacturer’s website.)

Remove the cover, and check for food buildup in the basket beneath. Use your hand or a spoon to remove any debris, replace the cover, and run the dishwashing cycle again.

Note: If you find a lot of food debris in the drain basket, prevent future clogs by pre-rinsing your dishes. Most dishwashers manufactured today feature macerators that grind bits of soft wet food, but they don’t have the power of a garbage disposal. Even if your dishwasher’s manual says you needn’t pre-rinse the dishes, do so anyway to avoid repeatedly cleaning the drain basket.

Check the dishwasher drain hose for kinks.

A dishwasher drain hose—the lightweight, ribbed plastic hose that connects from the dishwasher’s drain pump to the garbage disposal (or to an air cap)—can prevent water from draining if it is kinked. If something large or heavy was shoved under the sink it might have hampered the dishwasher hose, so explore the area and locate the hose. If it is kinked, try straightening it out manually.

Unfortunately, once a drain hose kinks, it tends to happen again in the same spot. If this continues to happen, replace the drain hose. (It’s a simple enough DIY task—keep studying for details.)

Examine the drain hose for clogs.

If the hose isn’t kinked, it could still be clogged with food sludge or debris. To check for a clog, you’ll need to remove the lower front panel of the dishwasher and locate the spot where the ribbed hose attaches to the drain pump.

First, unplug the dishwasher: While you shouldn’t come in contact with any wiring, the general rule of thumb when working on appliances is to unplug them first. Place old towels under the unit and remove the lower front panel. Many panels snap off but, depending on your model, you may have to remove a screw or two. Disconnect the hose from the pump (if you’re unsure where it is or how to disconnect it, consult your owner’s manual).

To check the hose for clogs, simply blow through it. If air won’t pass through, you’ve got a clog. If the clog is located at either end of the hose, you can try to carefully remove it with a screwdriver or other thin implement, like a straightened out wire coat hanger.

If the clog is not located near the end connected to the pump, remove the hose where it connects to the garbage disposal or air cap to check for a clog on that end. If the clog is lodged deeper, you’ll probably have to replace the entire hose. Don’t try to clear the clog with a plumber’s snake. Dishwasher hoses aren’t designed to withstand the tool’s cutting motion and can be easily punctured.

Replace the drain hose.

For recurring kinks or a clog that you can’t remove, you’ll have to replace the entire hose. Call a plumber if you’re not comfortable trying this yourself. A pro could easily charge a minimum of $150 for a house call, so you’d save a bundle making the repair yourself. First, you have to unplug the dishwasher and slide it out from under the countertop.

Next, disconnect the old hose from both the pump and the garbage disposal (or the air gap), and attach the new one in the same manner. Consult your owner’s manual first to ensure that you purchase the correct replacement hose, and for any specific instructions on how it attaches to your appliance.

RELATED: 5 Reasons Your Dishwasher Is Leaking (and How to Fix It)

Call the plumber.

When we’re faced with a dishwasher not draining, one of the above techniques will usually solve the problem. If you have tried all these tips to no avail, the problem could lie in a faulty dishwasher pump, or in the dishwasher’s timer or motherboard. Replacing these specialty parts should only be done by a licensed plumber, so if you’re in that situation, call a plumber.

Final Thoughts

Most of the time, when you find a dishwasher not draining, the remedy is simple and inexpensive. It’s likely the result of a full garbage disposal that won’t allow the drain water to enter from the machine, so it backs up in the bottom of the dishwasher. Barring that, the problem is probably a kink in the drain hose, or a clog in the hose, the drain, or the air gap. When determining how to drain the dishwasher, the homeowner rarely needs to call a plumber if they follow the above steps.

Depending on the brand of dishwasher, you’ll find that most machines don’t have macerators (grinding disposals). For this reason, putting dishes in the dishwasher that still have chunks of food is a sure recipe for a clogged drain. Rinse your plates before washing them!

FAQs  

Since it’s one of the handiest appliances in the home, the dishwasher is easily taken for granted—until it stops working. If dirty water is standing in the bottom of the machine after the cycle completes, the dishes will also be dingy, and the machine might smell bad. When the dishwasher won’t drain, a few questions are in order.

If the dishwasher won’t start, make sure the breaker that serves the electrical circuit hasn’t flipped off. Then, check to ensure the machine’s plug hasn’t become dislodged from the electrical outlet. Other things to check are whether a child safety lock has been activated and whether something is keeping the machine’s door from closing tightly.

Q: How do you clean a dishwasher drain hose without removing it? 

You might be able to clean a dishwasher drain hose by pouring 1 cup of baking soda and 1 cup of white vinegar in the machine’s drain and then letting it sit for up to 15 minutes before flushing with hot water and then running a rinse cycle. This usually only works for clogs in the hose near the drain.

Q: How do I get rid of standing water in the dishwasher?

Try running the disposal and rerunning the dishwasher cycle. If that doesn’t remove the water, bail it out manually and proceed with the troubleshooting sequence.

Q: Will Drano ruin a dishwasher?

It certainly could. Drano is caustic and can eat through a plastic drain hose and damage the machine’s motor.

Q: Can you put baking soda and vinegar in dishwashers?

Yes. Mix equal parts of baking soda and white vinegar and put the solution in the drain for a few minutes before running a cycle. When used regularly—every month or so—this solution can help keep clogs from forming.

C. Scott Brown / Android Authority

Does my phone, stereo, or car support Android Auto?

C. Scott Brown / Android Authority

Android Auto is not working on recently purchased phones

C. Scott Brown / Android Authority

Check Android Auto app settings

Edgar Cervantes / Android Authority

Android Auto problems with the USB cable

C. Scott Brown / Android Authority

Problems with Android Auto Wireless

C. Scott Brown / Android Authority

Fixed/known Android Auto problems

C. Scott Brown / Android Authority

While I normally sport the well-worn fedora of a hard-boiled sysadmin, Sunday mornings I swap that neo-noir accessory for the tech-noir: a pair of pro headphones. This is the tale of the collision of those two roles. An educational caper, dear reader. You see, my weekly gig is to run a Facebook Live Stream, and Facebook just recently began enforcing a new policy: all video streams are required to use encryption. We have Fedora installed on the media machine, and use Open Broadcaster Software (OBS) to stream. It should have been easy to update the stream settings. I made the necessary changes and tested it out — no luck. The error message was less than helpful: “Failed to connect to server”. With a sigh, I took off my headphones, put my sysadmin hat on, and walked out into the digital darkness. It was time to get back to work.

What the RTMPS?

Some terms, before we dive in. RTMP is the Real Time Messaging Protocol, originally developed by Macromedia. Thanks to Adobe, a version of RTMP is now an open specification, and many video streaming services now use it to transport live audio and video across the internet. RTMPS is simply the encrypted version, where RTMP is wrapped inside a TLS/SSL connection. TLS, Transport Layer Security, is the same protocol that powers HTTPS. TLS does depend on your machine having a good copy of the certificate bundle, a collection of public keys that are considered to be trustworthy.

How does one go about trying to fix this sort of problem? A good first step is to get a more useful error message. Running OBS from a command line lets us see all the extra output messages that are usually invisible. Below I’ve cut out all the extra messages, to highlight the failing connection attempt.

info: [rtmp stream: 'simple_stream'] Connecting to RTMP URL rtmps://rtmp-api.facebook.com:443/rtmp/...
info: RTMP_Connect1, TLS_Connect failed: -0x7680
info: [rtmp stream: 'simple_stream'] Connection to rtmps://rtmp-api.facebook.com:443/rtmp/ failed: -2
info: ==== Streaming Stop ================================================

Now we’re getting somewhere. The failure is specifically in the TLS connection, which we could have guessed. We also get an error code. Quick note on trying to search for that error: Google will interpret the leading dash as an indicator that you want results that don’t include that search term. Surrounding it in quotation marks, "-0x7680" is the way to get useful results.

How to Narrow Down the Cause of an Error

Searching for that error number will bring up two interesting hits. One is the thread in the OBS forum where those of us facing this problem have been discussing it. The other hit is the mbedtls documentation, where an error with this code is defined. It’s possible that it’s a false positive, but since we’re troubleshooting a TLS issue, it’s likely related. That error is MBEDTLS_ERR_SSL_CA_CHAIN_REQUIRED, and is described as “No CA Chain is set, but required to operate.”

So what’s next? We’ve learned a bit, but still don’t have any answers, so let’s dive into code. My standby quick-and-dirty debugging technique is to add printf() calls to help follow code execution, but where to start? We have a breadcrumb in the program output, “RTMP_Connect1”. Searching the OBS codebase on Github lands us at a function with that name. Partway through the function, we can see the command to print the log message that brought us here.

The error message indicates that a CA chain isn’t loaded. That sounds like an initialization problem. Perhaps that term “chain” is used in the OBS source. Searching our suspect file returns 18 hits, 17 of which are in a function named RTMP_TLS_LoadCerts(). I spent some time chasing down the execution flow of the RTMP connection, even making a sketch of when each function gets called. The code led me back to RTMP_TLS_LoadCerts(). That function has quite a bit of code in it, but we can safely ignore the parts that are specific to Windows or MacOS. There is an obvious line that should load system certs.

if (mbedtls_x509_crt_parse_path(chain, "/etc/ssl/certs/") &< 0) {
        goto error;
    }

So OBS makes a function call to the mbedtls library, requesting that the certificates in /etc/ssl/certs/ are loaded. Let’s make sure a proper certificate file is actually there where OBS expects it to be:

Ca-bundle.crt is the file we’re looking for. Notice the teal color? Those three files are actually symlinks to another location. Files on Linux filesystems are symbolically linked all the time, so likely not a problem. I spent some time checking things like file permissions, and tried disabling selinux, but came up with nothing. It didn’t seem to be an overzealous security setting. All I was left with was the knowledge that I had an mbedtls function that should be loading the certificate bundle, but when the program actually tried to verify a TLS certificate, it complained that the chain, ca-bundle.crt, was missing. If the mbedtls function was failing, shouldn’t it error out there? The next logical step is to look at the documentation for that function.

Now we find the first real hint about what could be happening. mbedtls_x509_crt_parse_path() can partially fail, and still give us a return code that doesn’t trigger an error. So, time to use printf() to see what that return code is on my machine. I added the code, compiled, ran the output binary… and got no such log output.

It took longer than I care to admit for me to figure out why my code changes didn’t seem to make a difference when running the program. This is a potential gotcha to watch out for. OBS uses a modular structure, consisting of the OBS binary, as well as various loadable modules. The code changes I was making were a part of obs-outputs.so, and even when running the compiled binary, those modules were being loaded from their default locations. To test my changes, I had to explicitly tell OBS to use the newly compiled module.

System Calls That Hate Symlinks

Something was obviously amiss with mbedtls_x509_crt_parse_path(). I wasn’t seeing anything obvious in the documentation, but I did see a similar function, mbedtls_x509_crt_parse_file(). What would happen if we forced mbedtls to only try loading the one crt file that we care about? I made the change, compiled, and to my surprise OBS finally connected to Facebook Live. I had a real fix, but I still didn’t understand why it was broken. It’s time to look at the mbedtls sourcecode.

The parse_path() function is easily found in the mbedtls source tree. Make sure to watch the #if defined() blocks — We’re not interested in the code for Windows. Once we find the loop that runs for each file in the given path, the problem code might jump out at you.

        else if( stat( entry_name, &sb ) == -1 )
        {
            ret = MBEDTLS_ERR_X509_FILE_IO_ERROR;
            goto cleanup;
        }

        if( !S_ISREG( sb.st_mode ) )
            continue;

Stat() is a system call that gets the status of a filesystem path, and then S_ISREG is a macro that checks whether that path is pointing at a regular file. Notably, the parse_file function does not do that check, and will happily load a symlink.

Report the Bug, But to Whom?

That, of course, is the core problem. Fedora uses a symlinked ca-bundle.crt, and mbedtls refuses to load ca-bundle.crt when it’s a symlink. We understand the problem, but what’s the proper fix? Which project actually has the bug here? OBS used the mbedtls function properly according to its documentation, and mbedtls may have a good security reason for refusing to load a bundle that’s actually a symlink. Is it on RPMFusion, and the package maintainer to fix the incompatibility? Personally, I think it’s really an mbedtls problem, particularly because this quirk isn’t mentioned in any documentation that I came across. Ultimately, it’s not my call which project needs to own this problem.

Our last task, then, is to report the bug we discovered. It’s a good idea to stop at this point, and ask yourself, is this bug a potential security issue? It’s best to try to report security issues privately, and most projects have contact instructions for disclosing those sorts of issues. Depending on where you found the problem, you may even be eligible for a bug bounty reward for finding the problem.

Assuming there is no security angle to consider, you’ll want to make a bug report. Does the project have a public bug tracker? That’s probably where it should go. If not, there is likely a mailing list where bugs are reported. Include enough information to reproduce the bug, and details on what you think is happening, but don’t include a bunch of log output in the bug or the mailing list. If it’s relevant, use pastebin or one of the other text hosting sites, to avoid including a wall of text in the bug report. If you have an idea of how to fix the problem, mention it. On a mailing list, patches are usually accepted. If the project is using Github or Gitlab, you can report the bug, and turn around and submit a pull request to fix it.

Particularly for trivial changes, I tend to ask what the project prefers, should I send a pull request, or is this trivial enough to fix without one. If you’re looking to do future work on the project, doing a PR is a handy way to get your name into the git record. Projects are more likely to look kindly on your future work, if there’s record of you already fixing bugs.

The End of Another Tale

This one turned out well enough. OBS is adding some workaround code to make sure the ca-bundle is properly loaded on systems where it’s a symlink. The mbedtls project sees this behavior as a bug, and I’ve submitted a patch to fix it. I noticed a related logic bug in the certificate loading code, and it’s been acknowledged as well. I’ve patched my copy of OBS so live-streams work again. It’s all in a days’ work for for the sysadmin. No rest for the weary, though. I have a pair of 10Gb Ethernet cards that die whenever they transfer VLAN tagged traffic. Just another case.

Errata

I know the more experienced programmers will point it out in the comments, stat() doesn’t ever set st_mode to S_IFLNK. Stat() follows the symlink to report on the target, while lstat() tells you about the symlink itself. My fix worked, but the problem was slightly different than I thought it was. Mbedtls_x509_crt_parse_path() can return a positive value if only some of the files in the specified directory successfully loaded. OBS was treating that positive value as a failure, and immediately dumping the certificates that had been loaded. Chasing false leads like this is totally par for the course when it comes to finding and fixing bugs. In the end, the bug is fixed, and that’s what really matters. Now if you’ll excuse me, I need to find something to wash the taste of crow out of my mouth.